#248654 - 09/02/2005 08:35
empeg file structures
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I am looking into creating my own empeg sync process - unfortunately mp3tofid doesn't quite fit the bill. But I am in need of info about the file formats and hope someone can help with some advice.
Is there somewhere I can find full and precise details of these formats?
Failing that:-
Does the tags ('1') file for a track HAVE to include all the tag info (my files have some different lines to what I've read elsewhere) and is the order important? e.g. is 'offset' necessary? The Rio Receiver defines this as a possible tag, but it is not supplied and hence is not required, but what about the RioCar (empeg)?
Likewise for a playlist tags file. If there is no appropriate artist etc, can this be simply left out?
What is the structure of the actual playlist ('0') file? It is not ascii (as are all the tags files). Are these stored in binary format? Any special requirements?
How can I create the /var/tags, playlists and database files?
tags appears to be ascii, but with a lot of blank lines. Do I need to touch this at all?
The database and playlists files appear to be binary, would they be dbm files by any chance (hope so:-). If not what?
Can anyone suggest how I can appropriately split the files across the 2 drives? Whatever method I use I need to be able to repeat the same split so that rsync will work efficiently. I thought I'd create 2 separate local fids dirs and simply rsync them, but what would make sense in deciding into which fids dir each file should be placed? I have some ideas, but hoped for some independent input.
Any help with this would be much appreciated?
|
|
Top
|
|
|
|
|
#248655 - 09/02/2005 12:31
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
The *1 tags files are not used at runtime by the player. The only reason they exist is for input when the player has to rebuild the binary database from scratch. The Hijack webserver also uses the *1 tags though, so it's a good idea to keep them around.
The *0 playlist files are just a series of 32-bit fids in binary form, in empeg byte order (little endian -- thanks Roger).
The database file format is just a catenation of most of the *1 tag fields for each fid, with separators etc.. easy to reverse engineer with a binary hex/ascii viewer/editor.
Cheers
Edited by mlord (09/02/2005 13:37)
|
|
Top
|
|
|
|
|
#248656 - 09/02/2005 13:02
Re: empeg file structures
[Re: mlord]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
in empeg byte order (big endian?).
Little-endian.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248657 - 09/02/2005 15:56
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I can easily create the 1 tags files, so might as well. Is the order of the lines important?
Excuse my ignorance, but in the playlist files, are the fids all full 32 bits, so it's like a fixed length record structure? How else could it determine the individual fids?
Is the /var/playlists file of the same format/structure as the database file? What about the /var/tags file. Is their structure fully explained anywhere? It would help enormously if I could short circuit the reverse engineering of these. Surely someone has this info written down somewhere?
I confess, trying to manipulate binary files like this is right at the limit of my knowledge so I need to leverage any help I can get:-)
|
|
Top
|
|
|
|
|
#248658 - 09/02/2005 16:42
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Surely someone has this info written down somewhere?
No. The only public documentation for the format of these files exists in the emptool source code.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248659 - 09/02/2005 18:28
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 12/02/2002
Posts: 2298
Loc: Berkeley, California
|
Perhaps you could tell us what you're trying to accomplish in the end? There are quite a few tools available that perhaps could be adapted to suit your needs more easily than building something from the ground up.
Matthew
|
|
Top
|
|
|
|
|
#248660 - 09/02/2005 20:04
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Yeah, the fids are all 32-bits in the playlist *0 files. You know how many there are by simply looking at the "length" tag in the *1 files, or just the Linux filesize in bytes.
The two files named playlists and tags are special: I don't remember what playlists is actually for, but tags gives the layout (field order) for each record in the database file -- it lists the tags that are actually known/used.
The database file then just has a record for each tune/playlist, with the variable-length fields in the order as given by tags, using NULLs (I think..) as field terminators.
Oh wait, I just looked at the database file, and it appears that each field within each record has two bytes in front: the first byte is the binary index of that field into lines in the tags file, and the second byte is the bytecount for the ASCII data that follows. So if "artist" is the fourth line in tags, then the artist data in the database for "Bare Naked Ladies" would have 0x03 0x11 immediately in front of the band's name, on each record within database that pertains to that band.
Cheers
Edited by mlord (09/02/2005 20:14)
|
|
Top
|
|
|
|
|
#248661 - 09/02/2005 20:07
Re: empeg file structures
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
MMm.. the playlists file is just another playlist: a bunch of 32-bit (binary) fid numbers, one after another, for all of the playlists that are visible under the first level Playlists menu on the player. I think.
Cheers
|
|
Top
|
|
|
|
|
#248662 - 09/02/2005 21:50
Re: empeg file structures
[Re: mlord]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
Yes, it's a concatenation of all playlists in fid order.
Pim
|
|
Top
|
|
|
|
|
#248663 - 09/02/2005 21:56
Re: empeg file structures
[Re: ukengb]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
Quote:
Can anyone suggest how I can appropriately split the files across the 2 drives?
When using (j)emplode, it's the player that decides on which drive a fid ends. The player knows the disk usage of both drives while the transfer is going on, so it probably takes the drive with the most space. This is as efficient as you can get.
mp3tofid cannot know the disk usage of the drives where the fids will end up, so it has to make a guess. You can specify the percentage of the number of fids that will be placed on the second drive. This does not have to correspond exactly with the percentage of disk space used, but it is close enough to allow you to fill both your drives to the limit.
Pim
|
|
Top
|
|
|
|
|
#248664 - 10/02/2005 10:46
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
As a NON C programmer, trying to read C source code is not a simple matter and there's lots of files in emptool. Can you be more specific about which files are worth looking at?
|
|
Top
|
|
|
|
|
#248665 - 10/02/2005 11:10
Re: empeg file structures
[Re: matthew_k]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Perhaps you could tell us what you're trying to accomplish in the end? There are quite a few tools available that perhaps could be adapted to suit your needs more easily than building something from the ground up.
OK.
I use solely iTunes to encode, maintain and control my music with the actual music data located on a linux server (a shared volume of). I then run a perl script (nightly) that reads the iTunes XML backup file and stores the relevant info in dbm files (actually uses DB_File) and this can be read and used by apache and some perl cgi scripts acting as the music server to several Rio Receivers around the house. Oh and I also use iTunes itself to play the music in my study/office.
So anything I do in iTunes is automatically available the following day in all the Rios (or earlier if I manually run the update) and it works perfectly.
Now I want to turn my hand to syncing the RioCar/empeg with the exact same music info (data and playlists) so that it all becomes a fully rounded iTunes/Rio music system solution, for me and anyone else who realises the benefits of using iTunes as their main control point.
None of the emplode type products comes close to doing what I want and are extraordinarily slow at doing what they can do. So I want to create a local fids structure and then use rsync.
Ah, "mp3tofid" I hear you say. Close, but no cigar. Very close in fact, but mp3tofid has to read the ID3 tags to get the data and I already have everything I need in the dbm databases, as well as the exact playlists I want to upload, which are already created and stored for use by the Rio Receivers. Actually writing the fids data will be relatively trivial and very fast, once I know what I need to write - hence my questions here.
As I've said elsewhere, I'm not a C programmer so trying to read C source files is a right PIA and in some cases I simply cannot make any sense out of them at all. Likewise trying to glean the file structures from sample files is also a non trivial task and prone to errors.
For these reasons I ask if there is any usable written definitions of these file formats to help me. Failing that I need to ask questions, but will ensure that the results of my quest are written up and made availabale for others.
However, in the meantime, I still need to know more about these files.
|
|
Top
|
|
|
|
|
#248666 - 11/02/2005 07:08
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 13/09/1999
Posts: 2401
Loc: Croatia
|
Hm, interesting approach. BTW, can you share your scripts that parse iTunes XML (I see (from WinAmp export) that the XML structure is quite straightforward, but I hate to rediscover the wheel (i.e. I am lazy  )) ? OK, I guess your best bet is to build all *0 (tune and playlist) and *1 (tag) files locally, transfer them over, remove database files (instead of bothering to recreate them) and let the player rebuild them from tag files. As already said, playlist *0 files are simply arrays of 32-bit numbers (member's FIDs), so they should be very simple to construct from perl of whatever (no C needed, but watch for endianness). AFAIR, tag files are pretty free-form: tags can be in any order, you can add your own (but they will not end up in the database).... Finally, maybe this is a good time to learn C? (says he, who is too lazy to parse a simple XML from Perl)  Cheers!
_________________________
Dragi "Bonzi" Raos
Q#5196
MkII #080000376, 18GB green
MkIIa #040103247, 60GB blue
|
|
Top
|
|
|
|
|
#248667 - 11/02/2005 07:09
Re: empeg file structures
[Re: pim]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
OK, I think I've got the file format/structures sorted, but I am puzzled as to the raison d'etre for the /var/playlists file. As Pim says, it seems to be a concatenation of actual playlist (0) fids, but why? What is its purpose? There's no delimiter separating each playlist's group of fids from the next group, so what exactly is it for?
Anyone enlighten me on this?
Oh, one more technical question on this. Does /var/playlists contain ALL the playlists (as Pim suggested) or just the top level (as I think Roger indicated) or some other combination?
Thanks.
|
|
Top
|
|
|
|
|
#248669 - 11/02/2005 08:57
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Does /var/playlists contain ALL the playlists (as Pim suggested)
No. Pim is right. /var/playlists contains ALL of the playlists. It's built by catenating the *0 files of each playlist, in FID order into one big file. It doesn't need delimiters because the length of each playlist is already in the database file.
It's an optimisation -- this way, the player doesn't need to load each *0 file when it starts up.
As Bonzi says, if you're just slapping a bunch of *0 and *1 files on the player, you can avoid generating anything in the /empeg/var directory, and let the player rebuild them itself.
_________________________
-- roger
|
|
Top
|
|
|
|
#248670 - 11/02/2005 17:28
Re: empeg file structures
[Re: ukengb]
|
journeyman
Registered: 08/08/2001
Posts: 51
Loc: CA, USA
|
Awesome project... I'd use (& test) it! Keep up the great work...
_________________________
Ryan here... Empeg [08000462] 40 Gig with Subaru WRX sport wagon attached... (still! pending memory + LED upgrade, sheesh, I've been sitting on my ass for years)
|
|
Top
|
|
|
|
|
#248671 - 12/02/2005 12:18
Re: empeg file structures
[Re: bonzi]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I want to finish this RioCar sync part of the project and get it all tidied up then make it generally available to anyone who wants. You're not in a rush are you?
My XML parsing is all my code - I don't use any other modules as I had very precise requirements for what it did. In fact these have changed slightly and become a bit more general, but I've already done it so pointless going back.
I originally ran the Rio Receivers on the ARM software, but have now moved to tRio which offers greater possibilities. Had to sort out a few glitches due to tRio doing some things slightly differently and Paul (tRio's author) sorted a couple of issues and so my system can now run with either Reciever player software.
With regard to the /var/??? files - it makes sense to me that I create them at the server end and then rsync the whole lot. Building the playlists and database doesn't look that hard and the tags file can be left as is I think.
I'll post back here when I have some results, may be a little while as I want to get another RioCar running to do the testing on and need a new HD for that.
BTW, I had thought of using the fids subdirs that Mark Lord makes use of for fidsift, but am I right in thinking that this only offers speed improvements in rebuilding the database? So if I do that on the server, there's no gain with using subdirs.
Can anyone comment on this?
|
|
Top
|
|
|
|
|
#248672 - 12/02/2005 12:29
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
Does /var/playlists contain ALL the playlists (as Pim suggested)
No. Pim is right. /var/playlists contains ALL of the playlists. It's built by catenating the *0 files of each playlist, in FID order into one big file. It doesn't need delimiters because the length of each playlist is already in the database file.
Of course. Juggling with lots of unknowns at present and that slipped my mind. Makes sense to me now. Does it HAVE to be constructed in that order (i.e. playlists in their fid order)? Or could the playlists be strung together in any order?
BTW, thanks for the tip about the database records containing an index into the tags file to specify the tag for the following data item. It would have taken me some time to figure that one.
Quote:
As Bonzi says, if you're just slapping a bunch of *0 and *1 files on the player, you can avoid generating anything in the /empeg/var directory, and let the player rebuild them itself.
I understand your point, but I like the idea of having everything ready on the server for a simple rsync, which can be relatively quick compared to a database rebuild. I always seem to want to sync the player last thing before I have to go out with it and then I get complained at for the delay:-(
So my theory is keep the local copy up to date (part of my nightly updates that read in the iTunes XML file) and be ready for 'Instant Sync'. In fact I think I'll copyright that phrase:-)
|
|
Top
|
|
|
|
|
#248673 - 12/02/2005 12:34
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Quote:
BTW, I had thought of using the fids subdirs that Mark Lord makes use of for fidsift, but am I right in thinking that this only offers speed improvements in rebuilding the database? So if I do that on the server, there's no gain with using subdirs.
The fids subdirs have a substantial, measurable impact (positive) on any and all operations dealing with the playlists, tracks, and tags files. They will help rsync as much as they help the internal database rebuild.
Many of us have 10000-20000 tracks on our players, for a total of perhaps 20000 to 40000 inodes in the /fids/ heirarchy. When those are all in a single directory, the clunky sequential search directory lookup mechanism (in the kernel) is very very slow, even just to play a selected track.
The fids subdirs really should be there by default in all software that accesses that data.
Cheers
|
|
Top
|
|
|
|
|
#248674 - 12/02/2005 12:59
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Does it HAVE to be constructed in that order (i.e. playlists in their fid order)?
Yes.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248675 - 12/02/2005 13:10
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
OK, here's a wild thought.
If the 1 files and the 0 playlist files (i.e. all fids except the mp3 files) are only used to build the /var/??? files and I will create those from my own data anyway, are they really needed? IOW can I get away with just the 0 fid music files and the /var/??? files and nothing else?
Seems that the player doesn't actually use anything else so why create them and have to sync them? This would really simplify the maintenance scripts and also save space on the empeg which is always to be desired.
Please tell me I'm right.
|
|
Top
|
|
|
|
|
#248676 - 12/02/2005 13:16
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
BTW, I had thought of using the fids subdirs that Mark Lord makes use of for fidsift, but am I right in thinking that this only offers speed improvements in rebuilding the database? So if I do that on the server, there's no gain with using subdirs.
The fids subdirs have a substantial, measurable impact (positive) on any and all operations dealing with the playlists, tracks, and tags files. They will help rsync as much as they help the internal database rebuild.
Many of us have 10000-20000 tracks on our players, for a total of perhaps 20000 to 40000 inodes in the /fids/ heirarchy. When those are all in a single directory, the clunky sequential search directory lookup mechanism (in the kernel) is very very slow, even just to play a selected track.
The fids subdirs really should be there by default in all software that accesses that data.
Cheers
I have about 13,000 songs/tunes/tracks at present so that applies to me. If you say it will impact on other activity (which you're right, is to be expected) then it makes sense to use the subdirs. So I will.
Thanks for the quick reply.
|
|
Top
|
|
|
|
|
#248677 - 12/02/2005 13:26
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
Does it HAVE to be constructed in that order (i.e. playlists in their fid order)?
Yes.
OK, thanks for that. Does the database need to be in a similar order? Since it's basically a concatenation of all the 1 tags files, do they need to be in fid order too? I assume so, but confirmation would be good to have.
|
|
Top
|
|
|
|
|
#248678 - 12/02/2005 14:14
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Does the database need to be in a similar order?
Yep.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248679 - 12/02/2005 14:16
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
IOW can I get away with just the 0 fid music files and the /var/??? files and nothing else?
No. When deciding whether to rebuild the database, the player scans the fids directories to see if the numbers tally. If they don't (and with your scheme, they won't), it'll force a database rebuild -- which won't work, because the files are missing.
So, in short, just put them there anyway -- they're minute when compared to the MP3 data, so don't worry about it.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248680 - 12/02/2005 15:49
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
OK, so I need the 1 tag files for the music data 0 files (mp3 etc), what about the playlists? Couldn't I just add the playlist data to /var/playlists and NOT have any 0 or 1 playlist fids? There would still be the correct ratio of 0 : 1 files (for the music data) and with NO playlist files at all, either 0 or 1 fids, wouldn't it be happy?
Just trying to pare it down to the bare essentials.
|
|
Top
|
|
|
|
|
#248681 - 12/02/2005 16:23
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
Quote:
When deciding whether to rebuild the database, the player scans the fids directories to see if the numbers tally.
If the database loads successfully, the directories aren't examined. There's no way the player stats 13,000 files on every startup -- it'd take minutes. I think the proposed cheat would actually work, but I'd still counsel against implementing it -- you'll confuse every other bit of car-player software out there, including Emplode and Jemplode.
Peter
|
|
Top
|
|
|
|
|
#248682 - 12/02/2005 17:16
Re: empeg file structures
[Re: peter]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
Agreed. Taking the simple extra step to preserve the tag files will allow the player to continue working normally with other software, thus providing a universal solution instead of a proprietary one.
|
|
Top
|
|
|
|
|
#248683 - 12/02/2005 22:03
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Yes, the tags and playlists are needed. The web interface requires them, as do some of the various add-on apps. Best just to keep them there, as they take up practically no space at all.
Cheers
|
|
Top
|
|
|
|
|
#248684 - 13/02/2005 05:52
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
OK, point taken.
Many years ago I was struck by something I read in Apple's "Human Interface Guidelines" (their guide to GUI design - everyone should read it). This was:-
"Consistency should be valued above idiosyncratic cleverness"
In this instance I guess I was just getting carried away being idiosyncratically clever.
Tags files are in:-)
|
|
Top
|
|
|
|
|
#248685 - 13/02/2005 06:01
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Soemthing else I need to consider - tags.
Specifically:-
rid - do I need to bother with this. Since the fid num. will be persistent and unique? Just use the fid?
fid_generation - is this the time it was created? Why would that be important?
ctime - of what?
bitrate - always preceded by 'vs' (Rio Receivers use 'fs'). Can do, but why the prefix?
I'm getting there. Thanks.
|
|
Top
|
|
|
|
|
#248686 - 13/02/2005 08:22
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Inspired by all the questions you're asking, I've started actually documenting the file structures. Go to http://www.differentpla.net/node/view/387 and take a look. If there's something I've not covered, or something that could be explained in more detail, post another comment here and I'll go back and update the document. Give me a short while to include all of the questions that have already been asked in this thread.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248687 - 13/02/2005 10:43
Re: empeg file structures
[Re: peter]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
There's no way the player stats 13,000 files on every startup
Fair point. ISTR that there's some kind of simple sanity check in there, though.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248688 - 13/02/2005 10:54
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
It doesn't stat() them, but it *does* read the fids directories and subdirectories, which gives it a list of what is present (at a fraction of the overhead of stat()).
-ml
|
|
Top
|
|
|
|
|
#248689 - 13/02/2005 11:03
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Good descriptions, Roger! Nice to see that our reverse-engineered view was mostly correct! I think the description there of the byte-ordering will confuse people, though.
cheers
|
|
Top
|
|
|
|
|
#248690 - 13/02/2005 11:42
Re: empeg file structures
[Re: mlord]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
I think the description there of the byte-ordering will confuse people, though.
Yeah, I wasn't particularly happy with how it came out, either. I've dropped that paragraph.
_________________________
-- roger
|
|
Top
|
|
|
|
#248691 - 13/02/2005 14:43
Re: empeg file structures
[Re: Roger]
|
pooh-bah
Registered: 13/09/1999
Posts: 2401
Loc: Croatia
|
Ah, excellent reference!
Mr FAQmeister, would you please put a link to it from some convenient place?
_________________________
Dragi "Bonzi" Raos
Q#5196
MkII #080000376, 18GB green
MkIIa #040103247, 60GB blue
|
|
Top
|
|
|
|
|
#248692 - 13/02/2005 15:34
Re: empeg file structures
[Re: bonzi]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
|
|
Top
|
|
|
|
|
#248693 - 13/02/2005 16:29
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
|
Damn, that's sweet.
Thanks for taking the time to put this together.
|
|
Top
|
|
|
|
|
#248694 - 13/02/2005 20:05
Re: empeg file structures
[Re: tfabris]
|
pooh-bah
Registered: 13/09/1999
Posts: 2401
Loc: Croatia
|
Heh, there was a discusion a few weeks ago about using Wiki for technical doc or similar site. Granted, Wiki can be handy, but having Tony around is much better! 
_________________________
Dragi "Bonzi" Raos
Q#5196
MkII #080000376, 18GB green
MkIIa #040103247, 60GB blue
|
|
Top
|
|
|
|
|
#248695 - 13/02/2005 20:28
Re: empeg file structures
[Re: tonyc]
|
addict
Registered: 02/04/2002
Posts: 691
|
I liked the notes on the rid. specifically the...
"Oh, and the R stands for Roger. Cheers."
_________________________
Oliver
mk1 30gb: 129 | mk2a 30gb: 040104126
|
|
Top
|
|
|
|
|
#248696 - 13/02/2005 21:41
Re: empeg file structures
[Re: oliver]
|
carpal tunnel
Registered: 24/12/2001
Posts: 5528
|
I thought it was Rio ID or something but now I know
|
|
Top
|
|
|
|
|
#248697 - 14/02/2005 07:28
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Well done Roger. Wish I'd been able to read that before figuring it all out for myself.
Couple of questions that relate to my project.
How important are the offset and bitrate tags?
Reason I ask is that the data I have readily to hand doesn't include the former and only the actual rate for tha latter, no indication of whether it's variable or fixed rate. I guess both can be obtained by looking at the actual file, but for speed I want to avoid that.
I know that the Rio Receiver doesn't use offset, even though it's a defined tag, it is not sent and currently ALL tracks are tagged by the server as 'fs', even though they're mostly 'vs' (I didn't write that bit), but it plays perfectly.
So can I get away with some innacuracy of the bitrate prefix and no offset specified? Is the RioCar as unconcerned about these as the Receiver seems to be?
|
|
Top
|
|
|
|
|
#248698 - 14/02/2005 08:18
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
How important are the offset and bitrate tags?
Not particularly. They're hints to the caching code, but the player ought to be able to cope without them. Just leave them out for your files and see what happens.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248699 - 14/02/2005 10:44
Re: empeg file structures
[Re: Roger]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
Implementing offset tags should prevent the player from loading id3 tags.
This is benificial if they contain album art or lyrics.
Pim
|
|
Top
|
|
|
|
|
#248700 - 14/02/2005 12:01
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
Quote:
Not particularly. They're hints to the caching code, but the player ought to be able to cope without them. Just leave them out for your files and see what happens.
Some releases do fail to cope if the bitrate is absent. None should fail to cope if it's inaccurate, though. If you're going to guess, it's more conservative to (a) overestimate and (b) go for vs rather than fs.
Peter
|
|
Top
|
|
|
|
|
#248701 - 15/02/2005 05:58
Re: empeg file structures
[Re: peter]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Guess I'll have to do some trial and error testing.
But, since I do know the bitrate, just not whether it's vs or fs, would it help to send just the numeric value and leave off the 'vs'/'fs' prefix?
|
|
Top
|
|
|
|
|
#248702 - 15/02/2005 07:30
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
No. It most definitely needs two characters before the number. The parsing's not particularly flexible. Just put 'vs'.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248703 - 11/03/2005 14:29
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I'm getting to the end of this, but have a slight problem. I am trying to arrange that nothing is required on the actual empeg in order to sync - all the control is from the PC/server side. So I want to run rsync as a daemon when on AC power, my sync script will then set the empeg read/write to do the sync and read only afterwards.
The trouble is that rsync will NOT start unless root is mounted read/write, but at the time that config.ini is run (and rsync would be started) of course, root is read only.
I can only think that I'll have to start rsync with a 'site exec rsync --daemon' via ftp from within the sync script, but is there another way to get the daemon running on startup?
|
|
Top
|
|
|
|
|
#248704 - 12/03/2005 19:04
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Still struggling with rsync since after the sync it stops the 'ro' command from working, which leaves some filesystems 'rw' and when I rebooted in this state it totally trashed the drives and had to rebuild from scratch. OK, that's why I'm using one designated 'test' empeg for this, but I am surprised.
What is the problem with leaving them rw when rebooting? What's the difference with a normal linux PC which normally runs rw?
Also, should I quit the player app while syncing?
When jEmplode syncs there's a neat graphic that displays and it 'locks the UI'. Is there any way I can do these with command line scripting?
|
|
Top
|
|
|
|
|
#248705 - 12/03/2005 20:15
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
A "normal PC" also mounts drives r/o before rebooting. This is "normally" done for you as part of the system shutdown/reboot scripts in /etc/init.d
Cheers
|
|
Top
|
|
|
|
|
#248706 - 13/03/2005 17:59
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Ah, that explains it. So it's entirely possible then that rebooting my system with it rw caused such catastrophic failure on my empeg?
I need to fiddle with rsync some more as it's very sensitive to things being rw or ro. It seems to be that specifying a log and/or pid file in rsyncd.conf prevents it starting when ro, but even without that, rsync can prevent the ro operation succeeding - keeping a file open maybe?
So is it OK to sync with the player app still running?
What about a nice graphic while syncing? Is it even possible to do this from the command line?
|
|
Top
|
|
|
|
|
#248707 - 13/03/2005 18:09
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
|
Quote:
What about a nice graphic while syncing? Is it even possible to do this from the command line?
pim's rsync binary supports a graphic progress indicator with the --empeg-progess switch. Do a forum search for mp3tofid and rsync and you'll find a script for handling the sync situation properly with respect to mounting drives, etc.
|
|
Top
|
|
|
|
|
#248708 - 13/03/2005 18:24
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
|
Quote:
I need to fiddle with rsync some more as it's very sensitive to things being rw or ro.
In the meantime you might want to consider using ext3 (see my sig for details) to decrease the likelihood of trashing your drives if you accidentally reboot when they are mounted rw again.
-Mike
|
|
Top
|
|
|
|
|
#248709 - 14/03/2005 07:44
Re: empeg file structures
[Re: mcomb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I will probably upgrade to ext3 once I know that my rsync routine is working correctly - I don't want to have to rely on ext3 to catch my errors.
However, I am a bit confused. Does Hijack now automatically contain the ext3 patches, or do you have to use the 'special' hijack, an older version which has had the relevant patches applied?
|
|
Top
|
|
|
|
|
#248710 - 14/03/2005 08:20
Re: empeg file structures
[Re: tonyc]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
What about a nice graphic while syncing? Is it even possible to do this from the command line?
pim's rsync binary[/url] supports a graphic progress indicator with the --empeg-progess switch. Do a forum search for mp3tofid and rsync and you'll find a script for handling the sync situation properly with respect to mounting drives, etc.
That's the rsync I'm using, but there are a couple of problems.
mp3tofid runs the empeg as the client and the linux server (with the music on) as the rsync server and although that's an option I'm exploring, I'm not getting any progress indication. I am capturing output of the rsync process so I can report on the outcome.
Would that prevent the indicator from working?
I did notice that there's a libresolv alongside the modified rsync, but as it already existed on the empeg I didn't install it.
Is that a different version required for the indicator to work? Pim?
The other issue is that I really want the empeg to be the rsync server (i.e. rsync running as a daemon) and control it all from the (linux) music server and if rsync is the server...
Would the progress indicator patch still work?
I suspect not.
I have been exploring the use of hijack's popup feature, but can't make that work either. I've tried a site exec via ftp or echo directly from the command line (via telnet) - the former returns an error and the latter does nothing.
What configuration is required to make hijack do this?
I've got [output]notify=1 in config.ini and I can read /proc/empeg_notify, but writing to it still does nothing. In desperation I added [input]notify=1 (seemed a logical idea:-), but that made no difference either.
I want to get the popup messaging working anyway, but ideally a graphic would be better during the transfer.
Can I somehow force a graphic to display?
As for setting the rw/ro etc, it's not a problem basically figuring when to do what, the problem is that rsync can cause the ro operation to fail (hanging onto a file probably) and that's what can be a disaster. Pim doesn't use a log or pid file on the empeg (in mp3tofid it's the client not the server). I wanted to make use of those files and indeed Roger uses them in his example of an empeg as an rsync server, but I have discovered this 'ro fail' problem. As I said, I need to play with rsync some more to ensure it does what it's supposed to do.
|
|
Top
|
|
|
|
|
#248711 - 14/03/2005 13:57
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
|
I guess I can't see the forest through your trees. What's so bad about having rsync run on the empeg as the client? Sounds like someone already solved 90% of the problem, and you're tossing that aside and inventing a new problem, but in the opposite direction. What needs are not met by having rsync do a "pull" from your music server instead of having the music server "push" files to rsyncd on the empeg?
|
|
Top
|
|
|
|
|
#248712 - 14/03/2005 15:18
Re: empeg file structures
[Re: tonyc]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
What's so bad about having rsync run on the empeg as the client? Sounds like someone already solved 90% of the problem, and you're tossing that aside and inventing a new problem, but in the opposite direction. What needs are not met by having rsync do a "pull" from your music server instead of having the music server "push" files to rsyncd on the empeg?
What I am doing here with rsync is merely part of a larger overall project that ties up an iTunes controlled music collection and makes it available to networked Rio Receivers and also can be easily sync'd to a RioCar/empeg. The basic concept is that all this is controlled on the main server (or with iTunes of course) and shouldn't involve messing around on the empeg itself - just connect it and tell the music server to sync, exactly as (j)Emplode do not expect/require any such interaction with the empeg itself.
Not only that, but Roger has implemented an rsync server on an empeg so I'm not the first to to attempt this.
You might imagine from this that what I am doing here is not the same as mp3tofid, e.g. I do NOT read the mp3 tags or the file folder structure that contains the mp3s. I have it all in the relevant databases and is much faster as a result. Hence mp3tofid provides somewhat less than the 90% you propose.
In any case, the problems I currently have are not really related to whether I'm pushing or pulling the data.
I'd like to get to the bottom of why the hacked rsync still doesn't display progress info.
Should I quit the player first? mp3tofid doesn't, but from what I've read it would appear to be a good idea. How can I do that without it re-starting?
I can't get hijack to popup messages. I've copied and pasted Mark's commands but it doesn't work. What else might need doing to enable this?
Is there any way to display a graphic on the empeg's display, i.e. just from the command line?
These are all issues that will help me to tidy up and tweak the UI of my stuff and help in sorting these issues would be much appreciated.
|
|
Top
|
|
|
|
|
#248713 - 14/03/2005 16:27
Re: empeg file structures
[Re: ukengb]
|
veteran
Registered: 21/01/2002
Posts: 1380
Loc: Erie, CO
|
Quote:
I can't get hijack to popup messages. I've copied and pasted Mark's commands but it doesn't work. What else might need doing to enable this?
echo 'popup 5 Test Message' > /proc/empeg_notify
or in a c program, open() /proc/empeg_notify and use write() to write "popup x message" to it.
Quote:
Is there any way to display a graphic on the empeg's display, i.e. just from the command line?
Not unless you write your own program to do this.
|
|
Top
|
|
|
|
|
#248714 - 14/03/2005 16:44
Re: empeg file structures
[Re: cushman]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Thanks for the info about displaying a graphic, even though it's not the answer I wanted, still at least I know.
I have overcome the display hurdles - almost.
The hijack popup command requires a time value or it's so quick you can't see it. The test example I copied (by Mark so I assumed it to be correct:-) did not include this. Once I realised this I can display messages.
The rsync progress display requires the different version of libresolv that can be obtained along with the hacked rsync. Once that has been installed the progress is displayed. Since this library already existed and the notes with the replacement version indicate it should be installed 'unless it's already there' I had thought I didn't need it.
A problem I have now found is that if a program (i.e. shell script) is started from the hijack menu (using ;@menuexec, the popup messages don't work - it appears menuexec takes exclusive control of the display which is a shame. Is there a getaround for this?
|
|
Top
|
|
|
|
|
#248715 - 14/03/2005 16:53
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
|
Quote:
However, I am a bit confused. Does Hijack now automatically contain the ext3 patches, or do you have to use the 'special' hijack, an older version which has had the relevant patches applied?
The hijack source includes the patches, but Mark's builds don't have them turned on so you either need to grab the kernel binary from my site or compile your own. Either way you'll want the newer versions of e2fsck and tune2fs from my site.
-Mike
|
|
Top
|
|
|
|
|
#248716 - 15/03/2005 09:16
Re: empeg file structures
[Re: mcomb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
However, I am a bit confused. Does Hijack now automatically contain the ext3 patches, or do you have to use the 'special' hijack, an older version which has had the relevant patches applied?
The hijack source includes the patches, but Mark's builds don't have them turned on so you either need to grab the kernel binary from my site or compile your own. Either way you'll want the newer versions of e2fsck and tune2fs from my site.
-Mike
I've grabbed e2fsck and tune2fs, but the trouble with using the hijack from your site is that it's not as up-to-date as the current build on Mark's site. I don't think it's practical to compile my own since I would guess that my regular RedHat linux tools wouldn't be suitable.
Any chance that Mark can simply turn on the ext3 support by default? Is thre any reason not to do this?
|
|
Top
|
|
|
|
|
#248717 - 15/03/2005 09:43
Re: empeg file structures
[Re: mcomb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
...you'll want the newer versions of e2fsck and tune2fs from my site.
-Mike
Forgot to ask this. Are the above utilities 'backwards compatible'? IOW can they be installed and used for the normal ext2 filesystems should ext3 not be used?
|
|
Top
|
|
|
|
|
#248718 - 15/03/2005 13:06
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Quote:
Any chance that Mark can simply turn on the ext3 support by default? Is thre any reason not to do this?
Yes, a very good reason: it takes up significanly more kernel memory. Kernel memory is non-pageable, so it becomes totally unavailable to userspace. Which means that the memory-strapped player software will have even less to work with.
Not an issue on my 32MB players, and not a really bad issue on my v2.0x equipped Mk2a players, but for anyone running v3 software and/or a Mk2 player (12MB instead of 16MB), that memory is NEEDED for other stuff.
Cheers
|
|
Top
|
|
|
|
|
#248719 - 16/03/2005 23:49
Re: empeg file structures
[Re: mlord]
|
pooh-bah
Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
|
Quote:
Yes, a very good reason
Yeah, what Mark said. If you don't know that you want to run ext3 and the consequences of using it you probably shouldn't have it forced on you. That said, I never had a problem running ext3 with the v3 betas on a stock mk2 with something like 10,000 mp3s so I'm not sure just how much extra memory it really uses.
Quote:
Are the above utilities 'backwards compatible'? IOW can they be installed and used for the normal ext2 filesystems should ext3 not be used?
Yep, they are just slightly more current versions of what is already on the empeg. They work fine with ext2 partitions.
Quote:
the trouble with using the hijack from your site is that it's not as up-to-date as the current build on Mark's site
I don't think anything significant has changed in the last few hijack builds which is why I never bothered updating, but it is probably time. Check my site in a half hour or so, I'll throw up a new version.
EDIT: Updated version (hijack 417) now available.
-Mike
Edited by mcomb (17/03/2005 00:16)
|
|
Top
|
|
|
|
|
#248720 - 24/03/2005 13:56
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Continuing in this thread as it's mainly to do with my rsyncing project.
For testing I have a small set of files locally that I can use to rsync with the empeg - about 20 mp3s and playlists - but the playlists and database files are simply the empty ones as originally created by jEmplode when there was no music at all. IOW, they do not correspond to the files that get uploaded.
After the sync the player app will not start, it just gives a lot of error messages of which this is the end:-
no room for private writable mapping
error: -12
no room for private writable mapping
error: -12
player(134): memory violation at pc=0x0207b864, lr=0x0203a954 (bad address=0x021
78014, code 2)
pc : [<0207b864>] lr : [<0203a954>]
sp : bfffba3c ip : bfffba60 fp : bfffba5c
r10: 2dc6cba0 r9 : 00002dc7 r8 : 00000000
r7 : 00008140 r6 : 0216fe38 r5 : 0216fe38 r4 : 00000178
r3 : 00001028 r2 : 02177fd4 r1 : 00000178 r0 : 0216fe38
Flags: Nzcv IRQs on FIQs on Mode USER_32 Segment user
Control: C0B8117D Table: C0B8117D DAC: 00000015
Function entered at [<0207b83c>] from [<0203a954>]
r6 = 00000178 r5 = 0216FE38 r4 = 02177F7C
Function entered at [<0203a868>] from [<0203aebc>]
r10 = 02175818 r9 = 02112374 r8 = 0214D808 r7 = BFFFFD88
r6 = 0214EF4C r5 = 02175808 r4 = BFFFFD80
Function entered at [<0203aea8>] from [<02061a54>]
The first line gets repeated a LOT first.
I thought that the player app would rebuild the appropriate files if there was a mismatch, why is it having such a problem here?
My intention is that the correct and matched playlist and database files will be uploaded at the same time with the music files and playlists, so this shouldn't be a problem in any case, but I want to get some idea of what's going on here.
Any suggestions as to what is the problem for the player?
|
|
Top
|
|
|
|
|
#248721 - 24/03/2005 14:21
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
no room for private writable mapping
error: -12
You're out of memory. -12 is ENOMEM.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248722 - 24/03/2005 14:53
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
no room for private writable mapping
error: -12
You're out of memory. -12 is ENOMEM.
I sort of guessed that, but why would it have such a problem with 6 playlists and 4 mp3 files (and their tags files too)?
Could it be down to the mismatch between the files and what is listed in the 'database' and 'playlists' files, or must it be a problem with the files themselves? As far as I can tell the files are structured correctly, but have I got en error in there somewhere?
What is likely to cause the player app to run out of memory?
|
|
Top
|
|
|
|
|
#248723 - 24/03/2005 15:57
Re: empeg file structures
[Re: Roger]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
In fact, it is just the subdirs in the fids folders that are causing the problem.
empeg:/empeg# ll fids?/
fids0/:
total 2
drwxrwsr-x 2 root 60 1024 Mar 7 01:07 _00000
drwxrwsr-x 2 root 60 1024 Mar 7 01:12 _0a2c2
fids1/:
total 2
drwxrwsr-x 2 root 60 1024 Mar 7 01:08 _00000
drwxrwsr-x 2 root 60 1024 Mar 7 01:12 _0a1c2
I am putting all the files into such subdirs just like fidsift does (at least I hope I'm doing it right) and with just these, i.e. with or without any files in them the player has this memory problem.
Is there something else that needs to be done to let the player know it should be using these 'fidsifted' directories?
|
|
Top
|
|
|
|
|
#248724 - 24/03/2005 16:09
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
Quote:
drwxrwsr-x 2 root 60 1024 Mar 7 01:12 _0a2c2
Woah, how big are your FID numbers? The car-player expects the FID numbers to be smallish; the database takes up one byte per unused FID below the maximum, and so the maximum wants to be pretty small. Optimum memory usage occurs when the FIDs are dense from 0x100 onwards.
Peter
|
|
Top
|
|
|
|
|
#248725 - 24/03/2005 22:11
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Those don't look like the expected naming structure for subdirs, either.
-ml
|
|
Top
|
|
|
|
|
#248726 - 25/03/2005 08:09
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Those don't look like the expected naming structure for subdirs, either.
-ml
Well, going by what you've said in the past, the fid (in hex) is padded to 8 digits with zeros to the left and the folder is then _xxxxx (first 5 digits) with the remaining 3 digits used for the filename.
Is that not correct?
|
|
Top
|
|
|
|
|
#248727 - 25/03/2005 08:22
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Is that not correct?
Quote:
_0a2c2
Well, that's fine, but you appear to have a FID numbered a2c2??0.
Assuming that this is the case, you've got something like 10 million unused FIDs. This is, as Peter says, 10Mb of RAM just to hold a single FF byte for each. There's no way that that's not gonna be a tight fit.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248728 - 25/03/2005 08:39
Re: empeg file structures
[Re: peter]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
drwxrwsr-x 2 root 60 1024 Mar 7 01:12 _0a2c2
Woah, how big are your FID numbers? The car-player expects the FID numbers to be smallish; the database takes up one byte per unused FID below the maximum, and so the maximum wants to be pretty small. Optimum memory usage occurs when the FIDs are dense from 0x100 onwards.
Ah, I may have missed an important factor here.
Are you saying that the fids really need to be contiguous (from 0x100) as the database file has to include the space for EVERY FID, whether it exists or not?
In that case I'm way off track. I start from 0x100 for the Master Playlist and then skip to 0x200, 300 etc for other top level playlists (0x640 max), then from 0x1000 for Art/Alb playlists (lots of them), then from nearly 2 million for the actual music files since the numbers are based on their inode, plus a constant (1 m.) to ensure there's no clash with the other numbers.
This means that there is no clash of numbers, but obviously there are big gaps in the range of fids, so if the database stores ALL the fids, then I'm being very wasteful.
A similar scheme works perfectly for my Rio Receiver music server, but that doesn't seem to care about discontiguous fid ranges, they're just ID numbers.
As you may remember, all my data is taken from iTunes which of course has its own numbering scheme for playlists and music data files, but irritatingly it has a tendency to re-number them, particularly after an iTunes update, so they're not much use for this purpose.
Are the problems with discontiguous fid ranges limited to the database?
|
|
Top
|
|
|
|
|
#248729 - 25/03/2005 09:45
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
Quote:
Are you saying that the fids really need to be contiguous (from 0x100) as the database file has to include the space for EVERY FID, whether it exists or not?
Yes. They don't have to be completely contiguous, but they need to be a lot less sparse than yours. The space taken up by a completely unused FID is only one byte, but if you've got several million unused FIDs that means you're still in trouble.
Quote:
Are the problems with discontiguous fid ranges limited to the database?
Yes. For instance, the Rio Karma, which has a rather smarter database, doesn't suffer from this problem.
Peter
|
|
Top
|
|
|
|
|
#248730 - 25/03/2005 10:07
Re: empeg file structures
[Re: peter]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Oh well, we live and learn. I hadn't appreciated this was the situation since the Rio Receiver is so different.
I'll just have to work on a new numbering scheme.
Thanks for the info.
|
|
Top
|
|
|
|
|
#248731 - 29/03/2005 14:28
Re: empeg file structures
[Re: peter]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
If there's no file(s) for a particular fid, does there have to be an entry in the database file? If so, what needs to be entered?
BTW, what's up with Roger's site? I want to re-acquaint myself with his explanation of the file structures, but it seems to be down.
|
|
Top
|
|
|
|
|
#248732 - 29/03/2005 16:45
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
Quote:
If there's no file(s) for a particular fid, does there have to be an entry in the database file? If so, what needs to be entered?
Entries must be present in the database for every consecutive fid from the smallest (which should be 0x100) to the largest. The database entry for a fid that has no files is just the terminating 0xFF byte.
Peter
|
|
Top
|
|
|
|
|
#248733 - 30/03/2005 06:53
Re: empeg file structures
[Re: peter]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
If there's no file(s) for a particular fid, does there have to be an entry in the database file? If so, what needs to be entered?
Entries must be present in the database for every consecutive fid from the smallest (which should be 0x100) to the largest. The database entry for a fid that has no files is just the terminating 0xFF byte.
Thanks. Figured it would be, but having missed a fairly major aspect of this, I thought I'd double check:-)
BTW, I still can't connect to Roger's site. Is it down? Anyone any news on that?
|
|
Top
|
|
|
|
|
#248734 - 30/03/2005 19:45
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 24/12/2001
Posts: 5528
|
It's hosted on his ADSL line at home.
|
|
Top
|
|
|
|
|
#248735 - 02/04/2005 12:17
Re: empeg file structures
[Re: tman]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
It's hosted on his ADSL line at home.
And it appears to have fallen over while I was away in the French Alps with no way to check on it. It's up now.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248736 - 02/04/2005 12:20
Re: empeg file structures
[Re: peter]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Entries must be present in the database for every consecutive fid from the smallest (which should be 0x100) to the largest.
Entries must be present in the database for every consecutive FID from 0x0 to the largest. The first entry (0x0) in the file has a type of "illegal", and the next 15 (0x10 to 0xF0) are a single 0xFF byte.
_________________________
-- roger
|
|
Top
|
|
|
|
#248737 - 02/04/2005 14:22
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
Quote:
Entries must be present in the database for every consecutive FID from 0x0 to the largest. The first entry (0x0) in the file has a type of "illegal", and the next 15 (0x10 to 0xF0) are a single 0xFF byte.
 Quite right. That's what happens with not rereading the emptool sources before posting.
Peter
|
|
Top
|
|
|
|
|
#248738 - 02/04/2005 16:05
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
Quote:
And it appears to have fallen over while I was away in the French Alps
How was the snow?
|
|
Top
|
|
|
|
|
#248739 - 02/04/2005 16:47
Re: empeg file structures
[Re: tfabris]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
How was the snow?
Pretty poor, to be honest. We had plenty of sun and some fresh snow on the last day, though.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248740 - 02/04/2005 16:58
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 24/12/2001
Posts: 5528
|
Quote:
Quote:
How was the snow?
Pretty poor, to be honest. We had plenty of sun and some fresh snow on the last day, though.
Meh. Still better than my skiing holiday in January where I injured myself on the first hour of the first day. I didn't even fall over! 
|
|
Top
|
|
|
|
|
#248741 - 20/04/2005 09:44
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I have re-written my uploader to utilise fids from 16 upwards with no gaps in the range, except due to deletions and these get re-used. So that all looks good now.
But I've still got a major issue.
If the music server is the rsync client so the empeg is the server and the player app is still running, then after the sync is finished I CANNOT set it ro. If I log in via telnet and try to set it ro it just reports "drive0 is busy". If I quit the player at the serial terminal and then exit so the player starts again, all is now OK.
I want to control the whole thing from the music server, i.e. have no interaction on the actual empeg and if I simply re-start the player app, it doesn't clear the 'drive0 is busy' error.
So, what might be causing drive0 to be busy and how can I circumvent that?
Should I sync while the player app is NOT running. This seems to work, but how can I stop it for the duration of the sync? Connecting via serial and quitting the app to start a shell session is not an option. As I said, I'm trying to get this to work without having to interact with the empeg.
Any ideas?
|
|
Top
|
|
|
|
|
#248742 - 20/04/2005 10:41
Re: empeg file structures
[Re: ukengb]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
if you have "killall" installed on your player, you could remotely invoke killall -INT player to stop the player, and killall -HUP bash to restart it.
Pim
|
|
Top
|
|
|
|
|
#248743 - 21/04/2005 06:48
Re: empeg file structures
[Re: pim]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
if you have "killall" installed on your player, you could remotely invoke killall -INT player to stop the player, and killall -HUP bash to restart it.
Have now:-)
However, last time I checked I found a thread where everyone was discussing how to stop the player, but the general consensus of opinion was that you couldn't as it would immediately re-spawn.
I'm pleased to have now got the solution. Thanks.
|
|
Top
|
|
|
|
|
#248744 - 21/04/2005 13:47
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Well, you needn't have killall installed on the empeg necessarily. It just automates finding the PID of the process(es). Sending signal 2 (INT) to the process, which is the same thing a Ctrl-C does, will end it without respawn.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#248745 - 21/04/2005 14:29
Re: empeg file structures
[Re: wfaulk]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Well, you needn't have killall installed on the empeg necessarily. It just automates finding the PID of the process(es). Sending signal 2 (INT) to the process, which is the same thing a Ctrl-C does, will end it without respawn.
It is tricky though without killall. The output from PS can be 'grep'd but you really need cut to get at the PID and cut isn't on the empeg by default so it's just as easy to get killall as cut (or anything you need to extract the PID).
|
|
Top
|
|
|
|
|
#248746 - 21/04/2005 14:45
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Oh yeah; I forgot that this was intended to be programatically called.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#248747 - 21/04/2005 19:29
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
No, you don't need cut. Nor "ps". You can grep through /proc/nn/ looking for the player, and use the bash built-in string processing to "cut" out the pid if need be.
Cheers
|
|
Top
|
|
|
|
|
#248749 - 22/04/2005 05:20
Re: empeg file structures
[Re: mlord]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
No, you don't need cut. Nor "ps". You can grep through /proc/nn/ looking for the player, and use the bash built-in string processing to "cut" out the pid if need be.
There are various other ways to achieve the same result, but they are either overly complex and/or require the installation of additional software and in that case I might as well install killall as that is intended to do and does EXACTLY what I want.
Mind you, I might put cut in there anyway, just in case I need it for anything else:-)
Thanks for the ideas.
|
|
Top
|
|
|
|
|
#248750 - 23/06/2005 10:53
Re: empeg file structures
[Re: peter]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Quote:
When deciding whether to rebuild the database, the player scans the fids directories to see if the numbers tally.
If the database loads successfully, the directories aren't examined. There's no way the player stats 13,000 files on every startup -- it'd take minutes. I think the proposed cheat would actually work, but I'd still counsel against implementing it -- you'll confuse every other bit of car-player software out there, including Emplode and Jemplode.
Peter
I may have some problems with my database file. The sync didn't complete (sorting that now), but the database file got uploaded, so the fact that some of the files are not there shouldn't cause a rebuild.
But.
Every time the player loads it rebuilds the database, yet the file itself is unchanged. When the player rebuilds the database, should it write the new data to the file, or just maintain it in memory (which means it'll rebuild again on next load)?
Under what circumstances would the player decide it needs to rebuild the database? What actual checks does it perform.
This would help me isolate why it is unhappy with my created file.
|
|
Top
|
|
|
|
|
#248751 - 23/06/2005 12:20
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 02/06/2000
Posts: 1996
Loc: Gothenburg, Sweden
|
Quote:
But.
Every time the player loads it rebuilds the database, yet the file itself is unchanged. When the player rebuilds the database, should it write the new data to the file, or just maintain it in memory (which means it'll rebuild again on next load)?
It justkeeps it in memory for that session - it can't write the database file to disk - because the disk is mounted read-only. IIRC the FAQ somewhere mentions how to get to the command line, remount the disk as read/write and manually start the player exe letting it write the file to disk .And then remount read-only so you don't get an fsck on next boot.
What little writing of data, playcounts etc, that does take place is to a raw partition - no filesystem...
_________________________
/Michael
|
|
Top
|
|
|
|
|
#248752 - 23/06/2005 12:32
Re: empeg file structures
[Re: mtempsch]
|
carpal tunnel
Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
|
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs
|
|
Top
|
|
|
|
|
#248754 - 23/06/2005 16:12
Re: empeg file structures
[Re: pgrzelak]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
Quote:
Do you mean this FAQ entry?
Well, sort of.
I build my database file directly, with my own code, so that's why there might be problems and I need to sort out the creation problem, rather than simply need the player to build one.
However, what I am now understanding is that when the player does build the database etc, it WILL save the files if the filesystem has been set rw, which answers my first question.
|
|
Top
|
|
|
|
|
#248755 - 26/06/2005 06:15
Re: empeg file structures
[Re: ukengb]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
The problem with my created database file is to do with 'extended' characters. E.g. the length of the following track title:-
Cú Chullain
is counted by the empeg player (and by perl on my Mac) as 11, but when I create the database file on my RedHat linux server, it counts the length as 12. So whenever one of this type of 'accented' characters appears it gets the length wrong. I have tried to not use anything except regular ASCII, but some names/titles really need them and it's not practical to strip them out and have to monitor every new track, just in case.
So I really need to find a solution and count the length consistently.
I know I could let the empeg create the database each time, but this is something I specifically want to avoid, so it's not a viable solution. I just want to make perl on the RedHat machine count the same number of chars.
Anyone any suggestions as to how to do this?
|
|
Top
|
|
|
|
|
#248756 - 26/06/2005 06:41
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 02/06/2000
Posts: 1996
Loc: Gothenburg, Sweden
|
Hmmm... what version of perl and how are you getting the length? If you have multiple accented characters, do you get 2 for each character or just a +1 error for the entire string?
_________________________
/Michael
|
|
Top
|
|
|
|
|
#248757 - 26/06/2005 07:37
Re: empeg file structures
[Re: mtempsch]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I get one extra count for each instance.
If I run this on the Mac (which counts the same as the empeg player)
perl -e '{print( "length = ", length "Cú Chullain","\n" );}'
it counts 11 chars, but the Intel RedHat machine counts 12.
If I modify that to start:-
perl -e '{use utf8; print....etc
Then RH perl counts 11 as I want, but the Mac gives an error:-
Malformed UTF-8 character (unexpected continuation byte 0x9c, with no preceding start byte)
Since I want this to be operable on both platforms, I still have a problem:-(
|
|
Top
|
|
|
|
|
#248758 - 26/06/2005 09:48
Re: empeg file structures
[Re: ukengb]
|
pooh-bah
Registered: 02/06/2000
Posts: 1996
Loc: Gothenburg, Sweden
|
Feels like a version and/or latformspecific bug... i386 Debian with an ancient perl (5.005_03) returns 11 for the plain (non utf8) statement (don't have utf8 pm installed.) Just toss another platform into the mix: Activestate perl 5.8.7.build813 on XP returns 11 for the non-utf8 and 7!! for the use utf8 case (had to muck a bit with the quotes and parens to get it accepted, this is what I ran: C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "{print(length 'Cú Chullain');}" 11) C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'Cú Chullain');}" 7 Inserting extra [normal] chars in various places in the string do get noticed: C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'Cú1 1Chullain');}" 9 C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'C1ú1 1Chullain');}" 10 But putting in the ú in various places it sometimes acts correct and adds 1, elsewhere I get a net negative change...and sometimes an error message. Can't see a pattern as to when/why it happens Some samples: C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'Cú Cúhullain');}" 8 C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú Cúhullain');}" 5 C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú Cúhuúllain');}" 6 C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú Cúhuúllainú');}" Malformed UTF-8 character (unexpected end of string) at -e line 1. 6 C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú CúhuúllainúA');}" Malformed UTF-8 character (unexpected end of string) at -e line 1. 6 Very funky... I don't recall how to do it, but I think one can make perl check version and libraries version at compile time, but don't recall if you can do a conditional use or not... beeen a while...
_________________________
/Michael
|
|
Top
|
|
|
|
|
#248759 - 26/06/2005 16:22
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
My guess would be that the Mac uses one byte to hold the ú, but RedHat is using two. My point being that your source file has the characters encoded differently, with RedHat using UTF8 and the Mac using some 1-byte codepage.
To test, create a plain text file on both systems that both contain the ú. Then do an 'od -x' on both of them and see how the character is encoded. My guess, again, is that it'll be one byte on the Mac and two bytes on the RedHat machine.
If that's the case, then your solution is to programmatically figure out what encoding scheme is being used "natively" on each system the perl script is being run on. I don't know how to do that, though.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#248760 - 26/06/2005 17:51
Re: empeg file structures
[Re: wfaulk]
|
member
Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
|
I'm sure you're right, it's a variation in the encoding schemes of each machine. What I orignally used to encode the info for the database file is:-
$tagsdb .= pack("U2A*",$tags{ $_ }, length $$hot{ $_ }, $$hot{ $_ } );
$tagsdb is then added to the database file - for each fid.
Now I preprocess the tag data like this:-
my $utag = $$hot{ $_ };
utf8::decode($utag);
$tagsdb .= pack("U2A*",$tags{ $_ }, length $utag, $$hot{ $_ } );
This seems to produce the correct length count on both the Mac and RedHat and most importantly when I rsync the resultant database file the empeg player no longer tries to rebuild. So that appears to be the solution to that problem.
I am still having trouble with rsync though. After the first bulk upload rsync crashed with an error I wasn't able to record and the recent update also produced a different error when trying to turn swapoff after the sync:-
swapoff: /dev/hdc6: Cannot allocate memory
but the sync did complete, although I'm sure it transferred far more then it ought.
The beauty of rsync of course is that you can simply try again and it catches up from where it left off so you just keep going till it shows no files were tranferred and then you know it finished. I just need to keep doing it to iron out these last probs, but I feel that with memory so short it may be impossible to entirely eliminate all issues.
So, it now seems to count ok, but any advice on running rsync in low memory situations (or with no terminal) would be welcome:-)
|
|
Top
|
|
|
|
|
#248761 - 27/06/2005 11:46
Re: empeg file structures
[Re: ukengb]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
...so the fact that some of the files are not there shouldn't cause a rebuild.
But it does -- the player checks that the count of FIDs on the disk matches the count in the database file. They don't match, so it'll rebuild.
As others have noted: the rebuild is in memory only. What's happening is that the player rebuilds the database in memory, and then attempts to write it to disk. Since the disk is read-only, it isn't written, and needs to be rebuilt next time. If (as the linked FAQ says), you have the disk mounted read-write when the player rebuilds the database, it will be written to disk, and won't need rebuilding next time.
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248762 - 27/06/2005 13:09
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Does it really actually try the open()/write() calls in this case, Roger?
If so, then perhaps I should have a Hijack GUI entry to permit it to succeed.. or perhaps have it pop-up a yes/no prompt.
??
|
|
Top
|
|
|
|
|
#248763 - 27/06/2005 14:30
Re: empeg file structures
[Re: mlord]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Does it really actually try the open()/write() calls in this case, Roger?
Yes, I believe that it does -- if open() fails for write, it just skips over the write(). Of course, it's been a long time since I looked at the code, so I could be mistaken.
Something in Hijack that allows it to be written would be cool -- presumably it'd remount rw underneath the player at this point?
_________________________
-- roger
|
|
Top
|
|
|
|
|
#248764 - 27/06/2005 14:39
Re: empeg file structures
[Re: Roger]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14493
Loc: Canada
|
Yes, that's the idea.. remount r/w during the call to "open()", and then remount r/o again on "close()".
Cheers
|
|
Top
|
|
|
|
|
#248765 - 27/06/2005 19:16
Re: empeg file structures
[Re: mlord]
|
enthusiast
Registered: 11/06/2003
Posts: 384
|
Quote:
Yes, that's the idea.. remount r/w during the call to "open()", and then remount r/o again on "close()".
That would be way cool. Thanks Marki!
--Nathan
|
|
Top
|
|
|
|
|
#248766 - 28/06/2005 06:34
Re: empeg file structures
[Re: mlord]
|
carpal tunnel
Registered: 18/01/2000
Posts: 5683
Loc: London, UK
|
Quote:
Yes, that's the idea.. remount r/w during the call to "open()", and then remount r/o again on "close()".
Sounds good -- bear in mind (and this is fairly obvious, so my apologies if you've already thought about it) that the player's writing 3 files (tags/playlists/database[3])at this point, so if you do prompt yes/no, it would be a good idea to only do it the once, lest user error cause the player to only write half of the needed information.
Unfortunately, I don't know the order that it writes them in.
_________________________
-- roger
|
|
Top
|
|
|
|
|
|
Previous Topic
Index