Hi,

I am upgrading my Empeg with a 160 GB harddisk after a long time being happy with my current setup containing 40 GB. Regarding this I have 2 questions. I will ask them at the end of this post. First some info.
- It is a Mark 2a Empeg, no memory upgrade
- There will be 2 drives in the Empeg, a 30 and 160 GB drive
- I have read the FAQ on disk upgrade issues and I know it by hand
- I performed a disk upgrade earlier (a small one, adding 2 20 GB disks)
- I searched the forum on disk upgrade issues
- I will install about 35.000 mp3's on the Empeg, in about 5000 playlists. The songs will be split in 2 parts at the root level, a roughly 20-80% song split
- I use random play over all my songs a lot, meaning that I would like to random play about 80% of 35.000 songs = 28.000 songs
- I will use Wendy flags (20) and filters (5)
- I will keep the amount of information in the tags as low as possible, only using the artist, title, album, year, and genre tags (all other tags cleared)
- I will (as per the faq) use the builder for big disks, the car2_v2.01_hijack.upgrade upgrade file, emplode 2.00, and the latest version of Hijack
- I will load the Empeg with 180 GB of songs in chunks of about 10-20 GB, resyncing after each chunk. I will do the whole from scratch, so I will start from a blank Empeg.
- I will do it via the ethernet connection, so it will take a long time. Shit happens, but it is not my time luckily

So, here are my 2 questions, the first one being the most vital to me.
1) I read on the forum that there might be an issue with the amount of songs (to be exact, the amount of fids) I am planning to use. This certainly holds for my random playlist desire. I also read that it may be beneficial to apply a 'hack' to the firmware (Patching the Player for more dynamic FIDs, the 'hack' called set_empeg_max_fid).
Question: since I will be using the car2_v2.01_hijack.upgrade upgrade file, is it still necessary to apply the 'hack'? Or is it already incorporated in the car2_v2.01_hijack.upgrade file?
2) Does someone have any good ideas or tips to enable me to random play as many songs as possible? Or in general, does anybody have any general tips or remarks concerning the upgrade process?

Many thanks for the answers,

Greetings from Jaap.

Hmmm - you've expressed yourself very clearly and it would be nice if you reported back on your experiences (here or on the wiki),

It's been a long time since I (and others I suspect) have actually done this kind of upgrade and ISTR that the FAQ doesn't address some of the big-disk issues? Of course I may be wrong and may have to buy Tony a drink at the meet but...

Are you doing it from Linux or windows?

I use Windows, but if needed I can use Linux. The windows computer is right next to the Empeg, so normally I use that one for song management.

I suspect, since the Empeg still is a very nice piece of work and drives are still getting cheaper and cheaper, that more people might be tempted to upgrade their player to larger capacities. May be, as in my case, for the last time (I hope at least since it takes a very long time to listen to 35.000 songs....2400 hours or so).
So, I am happy to write a description of the issues and solutions that I encounter and that are reported here. I am not a native english speaker, but I will do my best.

I don't know whether it's incorporated (I suspect not), but, unless Mark chimes in and says for sure, you should plan on applying the set_empeg_max_fid hack anyway. It doesn't do anything any harm to attempt to "re-hack" a player that's already "hacked".


The steps you describe are definitely the "normal" way to do the upgrade: the most straightforward and least error-prone. There are faster ways (in theory you can get the full disk bandwidth, upwards of 10Mbytes/sec, instead of 800Kbytes/sec over Ethernet or 400Kbytes/sec over USB1), but they're much more involved.

Peter

The set_empeg_max_fid.sh hack is *not* part of any version of the player software, so if you want its benefits you must apply it yourself. Prior to installing much music is best.

Note that there may be an even more optimal scheme for your setup than the defaults that this script employs.

If you want to improve upon it you'll have to read the BBS thread where it originated, in entirety. The basic script is intended to work on *any* player, populated or not, and has lower limits than it could have if "done right".

Also, the fidsift.sh hack is a *must* for your setup. But in this case you should probably apply it only after having added at least one tune + playlist to the machine (it's not been tested on a completely empty player).

Cheers

It's not the amount of FIDs exactly, it's the RAM those FIDS take up. So as you already surmised, one thing that will help is to carefully strip every file of any unnecessary tag data. For example, if you have the "Comments" field filled out on all of your songs, strip them all out now. When all that data gets concatenated together, that's your database file size. By the way, make sure to do your tag cleanup *before* attempting to install the songs onto the player.

For that large of a song set, though, doing this will only be of marginal help. But it will help some. All it's going to do is increase the number of songs that will work in a single shuffle, but you're still going to have to break things up into smaller groups even if you do that.



If you can figure out a way to split that into three groups of 33% each, that's going to be the most reliable way. Then you can reliably play one of those three groups, and the player will do all of the expected things, like remembering its song position when you restart your car.

I don't recall what happens if you try to shuffle-play all 35,000 songs, but I'm pretty sure one of those things is that it won't remember its play position or the shuffle order after a power cycle. If I recall, though, you're perfectly OK to shuffle the whole thing if you don't care about that feature and don't mind reshuffling after a power cycle.

Someone correct me if I'm wrong there.



If I'm recalling correctly, all this particular thing is going to do is to *slightly* increase the available size of the dynamic data partition, but it's not going to increase it to 35,000, so you still have to break up your playlist into groups that will fit inside this max_fid size (whatever that is). I don't think it's as big as 20,000 even, so I don't think your 80/20 split idea will work even with the max_fid hack. (Someone correct me if I'm wrong?)

No, Japie is correct: the number of songs that will work in a single shuffle (i.e. the problem with not remembering a large shuffled playlist over a reboot) is, indeed, to do with the amount of fids, not the total database file size.

Having a really large database is a different problem, as it takes up memory that would otherwise be used for caching. So a 35,000-song player would access its disks more often than one with fewer songs. If the database gets TOO big, there's no room for cache at all, and the player will fail to run. That limit is about 6Mbytes on a Mark 2a player; about 170 characters per song with 35,000 songs. It is much less on Mark 2 and Mark 1.


There's no need for smaller splits if using set_empeg_max_fid. With that, the 80%, the 20%, and even the 100% should all play fine.


Fortunately for Japie, I'm afraid you're wrong; with the set_empeg_max_fid hack, the "single shuffle remembered after reboot" limit is somewhere over 43,000 songs (depending a bit on how many playlists each track is in). Without the hack, the limit is about 21,000 songs.

The other limit on fid numbers, also ameliorated by set_empeg_max_fid, is the dynamic data size, used for storing play-counts and other, well, dynamic data. This is about 28,000 fids big by default; how big it becomes after set_empeg_max_fid depends on the exact disk geometry, but for big disks is typically about 44,000 fids. (That number includes playlists, though, so Japie is getting close on that one.)

Peter

This might be too geeky even for the FAQ, but FWIW here are the exact calculations.

A running-order will survive a reboot if it fits in the Bookmark Zero area of the dynamic partition. It is stored with eight bytes per distinct fid in the running-order, plus four bytes for each occurrence of each fid in the running-order. Or, alternatively, it's eight times the length of the shuffled playlist, plus four times the length of the unshuffled playlist. If each fid is in only one playlist, that's 12 bytes each. The Bookmark Zero area is 261,632 (0x3FE00) bytes long without set_empeg_max_fid, and 532,776 (0x7FE00) bytes long with it. So the absolute maximum is 261,632/12 or 21,802 songs as standard, 532,776/12 or 43,648 songs if hacked. If each song is in two different playlists (perhaps you have them organised by genre as well as artists), the hacked maximum is 532,776/16 or 32,736 songs.

The play-count storage size in the stock player is 28,672 fids long (one sector is used per fid). The size you get after using set_empeg_max_fid, though depends on the size of the dynamic-data partition. The builder image sets this at 16Mbytes, but this figure is rounded up to the nearest whole cylinder. Disks larger than 8Gbytes typically have a geometry with 255 heads and 63 sectors, so each cylinder is 255*63 sectors, that's 16,065 sectors, just under 8Mbytes. So 16Mbytes rounded up to the nearest cylinder is three cylinders, or 48,195 sectors. The first 4,096 sectors are used for other things (including the bookmarks), so that leaves 44,099 for dynamic track data. In this case, only songs -- not playlists -- ever have dynamic data, so in theory it'd be OK to have fids beyond the 44,099th as long as they were all playlists. However, no transfer software I'm aware of takes any care to give playlists high fid numbers and songs low ones, so you're likely to get in trouble as soon as you've got any fids beyond 44,099.

Peter

Oh, quit talking in generalities and give us some details! How can we figure things out if you won't provide us with information?

tanstaafl.

Okay, Peter, that is awesome data, thanks very much for correcting me, and sharing that information. I think I'll link this thread from the FAQ.

It'd be better to paste the information directly into the FAQ, but I haven't got the time to work on that at the moment.

Thanks again!

Impressive how quickly all this info can popup.

A short summary:
If you want to use large disks (say > 60 GB) there are some issues (some info from the faq, some from this and other threads):
- the standard disk builder is not working well. Use the builder for big disks instead (see the FAQ)
- idem for car2_v2.01_hijack.upgrade if you want to use the 2.01 firmware. There are also versions for the 3.x firmware (see the FAQ)
- if you want to random play many songs (for what many is see below) you have to apply the set_empeg_max_fid.sh 'hack' ( link ). Note that you have to apply this 'hack' each time you update/reload the firmware. In link you can read how to apply the 'hack' if you are not an experienced linux user
- keep the amount of info in the tags of the songs low. You can fill the regular fields (artist etc.), but do not add huge stories to e.g. the comment field. E.g. adding song lyrics is not a wise idea.
This also helps to keep the disk spinning up too often, preventing some noise (only relevant when used at home, not in the car)
- if you have many songs and get tired of the long database rebuilds, it is wise to apply the fidsift.sh 'hack' ( link ). Note that it is wise to apply this 'hack' after each upload of songs. There may be some issues with this 'hack' so read the whole thread
- there may be 'gaps' in the fids list when you delete and add songs quite often. This means that the maximum fid is higher than the counters mentioned below. So, to be sure you can actually reach the maximum, start from an empty disk and only add songs .

Now, what is a 'large' number of songs? This is dependent on the number of songs, the number of playlists per song, and the amount of information in the tags of the songs. There are 3 issues here:
1) The total number of a counter dependent on the number of fids. To do the math, a song counts as 2, the same song in a playlist as 1. So, if a song occurs in 2 playlists it counts as 2 + 1 + 1 = 4 etc.
The maximum number of the counter is 65408, 133194 when 'hacked'. This amounts to 21802 and 44398 songs (divide by 3) if each song is in exactly one playlist
2) The total amount of dynamic data, being the sum of playlists and songs. Dynamic data is dependent on the number of fids. This amounts to a maximum number of songs and playlists of 28,672, 44,099 when 'hacked'. The later number is dependent on the exact way the disk is manufactured, but for large drives in general the number is more or less acurate.
3) The size of the player database, being dependent mostly on the number of songs and the amount of information per song. If the database is too big there is no memory left to run the player software. On a mark 2a player the amount of memory available to the database is about 6 MB. This amounts to (6 MB divided by the number of songs) MB of data per song, which equals the average number of characters per song that can be in the tags. So, keeping the amount of data in the mp3 tags low helps to use a high number of songs.
Older players have less memory so can contain less songs. Does anybody have the exact numbers?

Given that you have a typical large disk of 160 GB with 35.000 songs and 3000 playlists comprised of albums containing the songs, you need the 'hacks' and it might fit:
- it has to fit on the disk ofcourse
- (2 x 35.000) + 3000 = 73.000, which is smaller than 133.197, larger than 65.408. You can even add additional songs or playlists. Note that each song has to be in a playlist since otherwise it can not be found on the player
- if you have a 'standard' disk, 35.000 + 3000 = 38.000, which is smaller than 44.099, larger than 28.672. So it will fit. However, this is dependent on the layout of the disk
- 6 MB / 35.000 = 172 characters per song in the tags. If you use the standard tags artist, title, album, track, genre, and year and clear all other tags you are safe since in general a regular song only consumes in this case half of this amount

Anybody any remarks on this summary? Thanks for the help and the feedback.

Jaap

The FID cap is altered by set_empeg_max_fid for your specific disk and how its partitioned. The total number of FIDs you're allowed isn't the same for everybody unless they've got exactly the same HD.

In a way, of course, it's slightly embarrassing that there are limits at all. But considering that the biggest Empegs ever shipped new had disks 10% the size of today's, and that most of these design decisions were taken in the Mark 1 era when disks were 2% the size of today's, I don't think we did too badly...

Peter

It's not quite as random as that. In fact, all the disks I've seen above 8Gbytes have the same cylinder size, so the same calculation would apply to them all. I'm just not sure whether or not that's actually guaranteed for every disk that big.

Peter

That bit isn't quite right. For this part, you need to count every time a song appears in a playlist; every song must appear in at least one playlist (or it can't be found), so your calculation should be at least (2 x 35000) + 35000. The conclusion is the same, though: fits with the hack, but not without it.

Peter

Thanks, Japie for revisiting and re-organizing the concepts behind big disk upgrades!

I'm pushing my two 60GB (111GB reported usable) drives more and more these days. I'm not sure how many files are on-board, but the 'big mix' of music and comedy is around 21,000 tracks. This does not include several thousand other tracks of books and lectures. Currently, Emplode reports about 12.7GB free space.

Anyway, I changed hard drives and operating systems on my home PC and lost my fidsift .bat file that tanstaafl put together. I figure any discussion about big disks should also mention that wonderfully helpful widget that he provided. So, here's a link to the thread Plain english description of what the "maxfid" patch does? in which the little bugger resides.

Also, I betcha it took Japie a while to put together some of his posts in this thread. No doubt the last upgrade to the board software helped him (or at least, didn't hinder him).
Cheers Japie, Mark, tanstaafl, and Drakino!

Heck, thanks to all you guys!

edit: just downloading more files and will try the fidsift.bat on my newer system in a moment. Also, I grabbed up the latest fidsift.sh file that Mark provided (299723) and patched it into the fidsifter directory for the .bat file, replacing (austensibly) 289080.

Nevermind...answered my own question.

I finally managed to finalize the upload of all the songs, and everything is working fine .
Some issues and remarks I encountered:
- I have 37.000 fids, everything is working fine
- the average upload speed via ethernet is only 0.45 MBps (including databse rebuilds etc.), a bit slower than I expected for an ethernet connection. It took me over 4 days to transfer all the songs
- running set_empeg_max_fid.sh is easy and quick, resulting in my case in the standard 44.000 (or so) fid-limit number
- running fidsift is best done via the serial port, not via ftp since it takes a very long time (couple of hours) to run it if you add 40 GB of songs
- fidsift gives a huge decrease in the database rebuild times. A decrease of 90% can be expected when you add 40 GB of songs
- Emplode is not very stable when you add 40 GB of songs. I encountered regular crashes on the transfer, resulting in very long (3 hours and more) database rebuilds and disk checks. JEmplode is more stable.

450 kilobytes/sec is about right for an ethernet upload. I don't think I've ever seen any more than about 600kB/sec (And that was probably reading). 10Mb Ethernet has a theoretical max speed of only about 1000kB/sec. The empeg also doesn't have DMA so disk reading and writing isn't the fastest.