It's not you, but rather the frickin closed-source firmware and Windows installer program. They insist on destroying a perfectly good partition layout, even when *not* running a builder image. I'm just using a bit of guesswork to try and prevent that.


Yeah, I'd really love to have some serial port messages, except the stupid software won't work over anything other than pathetically slow serial -- even on Linux we at least have an ethernet-based upgrader (open-source, too!) that runs at perhaps 100X that speed.

I really don't know why I persist in wasting my time to help Windows users with software that I don't need/use here. Stupid me, I suppose.

As for the "prevented 48258" message, there was also a "prevented 0" message too, which goes by so quickly you don't have time to read it. And hopefully also a "prevented 63" message.

Sector 0 is the master partition table, sector 63 is the linked-list entry for hda5, and sector 48258 on your disk is (0 + 63 + 63 + 63 + (2 * 24034) + 1), which was supposed to be the partition table linked-list entry for the swap partition, hda6. But I may still not have that one correct. EDIT: Yeah, looks off by that final +1. Fixed.

If it manages to prevent all three of those, then the partition table layout should not change.

I guess it didn't manage it. I'll try and beef up the messages some more, though that's very difficult to achieve here.

I'll respin it again (V7), with a couple more messages. This time, run the builder (I'll remove the safeguard, so you won't need to manually un-build the drive first), then run the software .upgrader, and watch carefully once the empeg logo appears on the screen (after the kernel is reflashed).

Write down the numbers you see, and post them here, along with the output of "fdisk -l /dev/hda".

That should be enough info for me to nail it properly next time around (or we might get it this time..).

Cheers

Did your earlier correct guess that it's the CHUNK_PUMPHDA (0x20) in the upgrade file that tells the firmware to do the re-partition operation, not work out? Omitting that would seem a more robust solution than preventing partition table rewrites on a sector-by-sector basis, particular if the actual sector numbers can vary with disk geometry.

And though most of the Windows installer is closed-source, the code it uses to do the upgrading is actually visible in the Emptool open-source distribution, lib/protocol/upgrader.cpp.

I still also think that you should arrange that, whatever the disk geometry, the music partition ends up at the same sector offset it would with the stock image. And that the easiest way to achieve that would be to repurpose, not resize, the existing partitions.

Peter

Here is what your disk probably looks like with bigdisk:

Code:

Disk /dev/hda: 255 heads, 63 sectors, 16709 cylinders
Units = cylinders of 16065 * 512 bytes

Device    Start     End    Blocks    Id  System
/dev/hda1     1      11     88326     5  Extended
/dev/hda2    12      16     40162+   83  Linux
/dev/hda3    17      21     40162+   10  OPUS
/dev/hda4    22   16709 134110620    83  Linux
/dev/hda5     1       3     24034+   83  Linux
/dev/hda6     4       5     64228+   82  Linux swap

Or, viewed differently:

Nr AF  Hd Sec  Cyl  Hd Sec  Cyl    Start     Size ID
 1 00   1   1    0 254  63   10       63   176652 05
 2 00   0   1   11 254  63   15   176715    80325 83
 3 00   0   1   16 254  63   20   257040    80325 10
 4 00   0   1   21 254  63 1023   337365268092720 83
 5 00   2   1    0 254  63    2       63    48069 83
 6 00   1   1    3 254  63   10       63   128457 82

The master partition table is always at sector 0.
The first extended partition list entry is always at sector 63.

But where is the second one, for the swap partition?
It will be at (63 + 48069 + 63) = 48195.

You can verify this, after running the bigdisk_builder (but before installing the software), using this command:
/bin/read_sector.sh 48195

The last line of output should end in 55 aa.

And sure enough, my off by one error in the last attempt would have missed preventing that from being overwritten..

So the current v8 should actually work, I think.

Cheers

No, that would only work with the new images. But I'm also trying to prevent a major mishap should somebody get confused and try to install an older image. Much more robust this way. It just failed on the last attempts because I didn't have the math entirely correct. What it's doing now is rather simple: disallow upgrader writes outside of the existing /dev/hda5 partition. Period. It's just taking a long time because I don't run MS Windows here, so I cannot actually test it very easily.



Nope. That doesn't help us. The current spare partition is still too small for 64MB of swap, and some people already use it as an adjunct to the root partition. So it stays as-is, and will not be" repurposed". And the dynamic data partition also could use some growing space to handle the larger running orders that may appear on these new huge drives.

The alternative is to use a swap file, rather than a swap partition, and to place that file onto /dev/hda4. Except I remember that 2.2.xx kernels happen to perform much better when swapping to a partition rather than a file (fixed in newer kernels).

There's nothing sacred about the factory layout that has long since been abandoned -- there'll never be another factory load to worry about, so we can simply use whatever new format we like. With safeguards to prevent accidents when somebody uses an ancient factory build.

Cheers

True, but largely irrelevant as it would give a total of 48MB (in fact, due to round-up-to-cylinder, actually nearer 56MB), which is a big improvement on 16.

Not on brand-new disks they haven't built before, they don't. I don't know of any third-party add-ons that demand to live on the spare partition and not, as an alternative, the music partition.

Again true, but largely irrelevant as you've already solved that problem with set-empeg-max-fid, which on typical large disks increases the size to nearly 24MB.

I respectfully suggest that putting a swapfile on /dev/hda4 in order to have enough swap to run fsck on /dev/hda4 is in itself not a great idea

Peter

Our "ancient" factory builds reflash the kernel before they pump the drive. (Who would have thought the old man had so much blood in him?) There's no safeguard you can put in Hijack to prevent that, because Hijack isn't running at that point.

Peter

Because you are AWESOME, and because you're essentially the only person on the planet with the skills, tools, and knowledge to do it right now.

Do you realize that if it weren't for you, most of the empegs out there would be paperweights by now? You're almost singlehandedly keeping this community going by doing things like this. I'm proud to know you.

Heh. Yeah, that too!

But I just remembered the real reason for wanting to hack the kernel here: I'm getting rid of the builder image altogether. There's no reason that a (new) oridinary player .upgrade file couldn't include builder functionality, except for this tricky little problem.

Cheers

FWIW, it was originally done that way out of sheer paranoia. Ensuring that "normal, everyday" upgrade files weren't capable of wiping the user's music by formatting the partition, helped reassure us that we wouldn't accidentally trigger that outcome.

Peter

Okay, there's a fresh set of v9 installers on the site now. These worked for me here a moment ago, and I'm rerunning them again now just for background pleasure.

The rather simple workaround is to block all early writes to the raw disk device /dev/hda, unless it lacks a valid empeg partition layout, and allow everything else to pass through.

Nice cool feedback on the "pumping" now, too. One can even see where the upgrader suppresses sending long streams of zeros over the serial link.

That'll do it for now.

-ml

Okay, they worked for me.

You'll have to nuke /drive0/fids/ again before trying them, though.

Cheers

There were good reasons for this; ISTR they are to do with the player running mlock'ed to prevent any non-explicit disk access that the kernel might think is a good idea when the disks are spun down (which then causes blocking at kernel level, as opposed to us explicitly managing the spin up/down in a disk manager thread). JohnR or Mike can probably elaborate on this.

The actual data volume is pretty small in terms of copy overhead so we didn't delve further into the 2.2 kernel to try and fix this.

Hugo

Random fact: the "pumping" term comes from how Acorn used to restore their R140 unix boxes in case of being trashed...

No complaints on the day -- your little mp3 player is a great machine, Hugo!

But for future designs.. mmap(). It can even play friendly with mlock/mlockall().

Cheers

Hi,

I had to work late tonight, couldn't get to this until now.

I installed using the same procedure tonight and this is what I got after the v9 builder...


bash-2.03# cat /proc/partitions
major minor #blocks name

3 0 244198584 hda
3 1 1 hda1
3 2 40162 hda2
3 3 40162 hda3
3 4 244027350 hda4
3 5 24034 hda5
3 6 64228 hda6
bash-2.03#


This is what I got after the v9 player software....


empeg:/empeg/bin# cat /proc/partitions
major minor #blocks name

3 0 244198584 hda
3 1 1 hda1
3 2 40162 hda2
3 3 40162 hda3
3 4 244027350 hda4
3 5 24034 hda5
3 6 64228 hda6
empeg:/empeg/bin#


!!!!!!!!!!! Looks Good !!!!!!!!!!!

Emplode sees it, it has the correct storage capacity displayed (232 GB).

I will load 20 GB onto it and see what it does.

Always wondered what the term "pumping" meant. I just paralleled it is the DOS world of formatting. Thanks for the clarification.

I certainly appreciate you supporting us poor saps that are stuck with Windows. I was playing with Windows when it was Windows 286 (really archaic) and it has definitely improved since then <grin>. I also tested & qualified UNIX V and SCO UNIX products Graphics drivers. Linux is really looking better every day.

The Empeg wouldn't have near the power without the well-thought-out software and hijacks you participate in the continual development of. You have worked hard on the big disk builders for me and the community. Thanks!

Definitely like having the display messages - they give a hint as to what's going on !!!! They do fly by very quickly and I first saw 02 then 62, blur, blur, then the pumping of hda5.

I'll let you know how it works out.

Thanks again,

Ross

Yeah, somewhat useful, I believe. And there's still just enough room on the line for a %complete indicator, too.

I may add that when I get around to fixing the errant pump message that happens on each boot, as the root filesystem is mounted (and written to.. wonder what's up with that?).

Hi,

It completed loading 22 GB of songs without error and everything looks right.

I will load another 30 GB of songs on it and see if it looks good.

So far, So good....

I'll keep you informed.

Thanks alot,

Ross

Hi,

Looks like we have a winner here (at least as far as I have tested it)!!!!!

It loaded 54 GB of songs (3428 songs, 113 Albums, 283 Playlists) and worked correcly. Most of these are .wav the rest are .MP-3s.

Do I need to install the FIDS patch still or was that part of the drive build/player software load?

Throughout the next few days (maybe weeks), I will load it up, purchase another 250 GB drive, and load around 400 GB of my CD collection on it. Can the patch be installed at anytime in the process? I will probably erase what is loaded and start anew, so if I need to apply the patch before loading, it's no big deal.

I really need to get my Ethernet up before I load that much onto the drives. Just no time to work on it yet.

I'll let you know if I run into any snags along the way.

You did good Mark!!!

Thanks again,

Ross

The program to do the patch should already be in /bin on the software load, but you do still have to run it (once). I didn't get round to automating that part (and it is disk drive specific, so it cannot be done in advance).


Earliest is best, like before any music is installed. But I've used it on my (pre-loaded) players here without ill-effects.

Cheers

Hi,

Okay, I will start from scratch and install the FIDS patch before music load. The patch needs top be run every time a build and player software load is performed since it is drive specific, right?

BTW, how large of drives will the new build/player software accomodate? How large for the FIDS patch?

Thanks,

Ross

I wouldn't bother reloading the stuff that's already there -- it should be fine. But if you build other new disks from scratch, then just run it earlier on those.

The patch just need be installed once after software installation. If you ever reinstall the software, then reinstall the patch.

If you later want to duplicate your drive for a second player (with identical drive type), then just clone the drive, and no patching is required if this is done as a whole drive copy.

Cheers

Hi,

That's what I will do then.

I will be duplicating the drives for multiple RIOs in my cars and around the house.

I thought the ones around the house can be a larger 3.5 inch drive (with additional cooling, larger power converters, possible separate power for the drives - common grounded of course, chassis tray modifications, etc)

That's why I asked about how large of drives were possible with the new build.

Thanks,

Ross

Receiver Edition? (but beware this)

Peter

Hi,

Been busy, I will check this out too.

Does it handle .wav's?

Thanks,

Ross

you can't stream .wav with the kind of network connection that the receiver and the empeg have. Not fast enough.

On real Receivers, you're right; while it's impressive that that HomePNA stuff worked at all, it never worked as well as it was supposed to. (Receivers have Ethernet too, which is certainly capable of streaming WAV, but we thought HomePNA users would be in the majority, and didn't want to make them second-class citizens.) For Receiver Edition, it works fine, although you need a server that's prepared to give you the WAV files in the first place. Most don't, as they're intended for use with real Receivers which can't "decode" the WAV format even if they could stream it fast enough. With a custom server, I've decoded Freeview DVB-T radio on the server and sent it as WAV to a Receiver Edition car-player.

If you're still worried about WAV bandwidth, the Receiver Edition is a v3 build, so you could send it FLAC instead.

Peter

I have 2 160 GB disks I want to install in my Empeg. Will the new disk builder setup both disks at once or will I need to do the disks one at a time?

Hi,

I only did mine a disk at a time to make them both bootable if necessary. Maybe the new version already does that, I didn't try it.

I haven't tried it with both drives, but I think Mark was trying to do that if you look at one of his previous responses in this thread.

I could try it with my 250 GB drives if you need me to...

Ross

No, I will try it myself this weekend.

It should do both of them.

Cheers