Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 4 of 4 < 1 2 3 4
Topic Options
#248744 - 21/04/2005 13:47 Re: empeg file structures [Re: ukengb]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Well, you needn't have killall installed on the empeg necessarily. It just automates finding the PID of the process(es). Sending signal 2 (INT) to the process, which is the same thing a Ctrl-C does, will end it without respawn.
_________________________
Bitt Faulk

Top
#248745 - 21/04/2005 14:29 Re: empeg file structures [Re: wfaulk]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
Quote:
Well, you needn't have killall installed on the empeg necessarily. It just automates finding the PID of the process(es). Sending signal 2 (INT) to the process, which is the same thing a Ctrl-C does, will end it without respawn.


It is tricky though without killall. The output from PS can be 'grep'd but you really need cut to get at the PID and cut isn't on the empeg by default so it's just as easy to get killall as cut (or anything you need to extract the PID).

Top
#248746 - 21/04/2005 14:45 Re: empeg file structures [Re: ukengb]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Oh yeah; I forgot that this was intended to be programatically called.
_________________________
Bitt Faulk

Top
#248747 - 21/04/2005 19:29 Re: empeg file structures [Re: ukengb]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14494
Loc: Canada
No, you don't need cut. Nor "ps". You can grep through /proc/nn/ looking for the player, and use the bash built-in string processing to "cut" out the pid if need be.

Cheers

Top
#248748 - 22/04/2005 00:05 Re: empeg file structures [Re: mlord]
JBjorgen
carpal tunnel

Registered: 19/01/2002
Posts: 3584
Loc: Columbus, OH
Quote:
No, you don't need cut

Maybe that should be phrased "I don't need cut." Some of us are mere mortals.
_________________________
~ John

Top
#248749 - 22/04/2005 05:20 Re: empeg file structures [Re: mlord]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
Quote:
No, you don't need cut. Nor "ps". You can grep through /proc/nn/ looking for the player, and use the bash built-in string processing to "cut" out the pid if need be.


There are various other ways to achieve the same result, but they are either overly complex and/or require the installation of additional software and in that case I might as well install killall as that is intended to do and does EXACTLY what I want.

Mind you, I might put cut in there anyway, just in case I need it for anything else:-)

Thanks for the ideas.

Top
#248750 - 23/06/2005 10:53 Re: empeg file structures [Re: peter]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
Quote:
Quote:
When deciding whether to rebuild the database, the player scans the fids directories to see if the numbers tally.

If the database loads successfully, the directories aren't examined. There's no way the player stats 13,000 files on every startup -- it'd take minutes. I think the proposed cheat would actually work, but I'd still counsel against implementing it -- you'll confuse every other bit of car-player software out there, including Emplode and Jemplode.

Peter


I may have some problems with my database file. The sync didn't complete (sorting that now), but the database file got uploaded, so the fact that some of the files are not there shouldn't cause a rebuild.

But.

Every time the player loads it rebuilds the database, yet the file itself is unchanged. When the player rebuilds the database, should it write the new data to the file, or just maintain it in memory (which means it'll rebuild again on next load)?

Under what circumstances would the player decide it needs to rebuild the database? What actual checks does it perform.

This would help me isolate why it is unhappy with my created file.

Top
#248751 - 23/06/2005 12:20 Re: empeg file structures [Re: ukengb]
mtempsch
pooh-bah

Registered: 02/06/2000
Posts: 1996
Loc: Gothenburg, Sweden
Quote:
But.

Every time the player loads it rebuilds the database, yet the file itself is unchanged. When the player rebuilds the database, should it write the new data to the file, or just maintain it in memory (which means it'll rebuild again on next load)?



It justkeeps it in memory for that session - it can't write the database file to disk - because the disk is mounted read-only. IIRC the FAQ somewhere mentions how to get to the command line, remount the disk as read/write and manually start the player exe letting it write the file to disk .And then remount read-only so you don't get an fsck on next boot.
What little writing of data, playcounts etc, that does take place is to a raw partition - no filesystem...
_________________________
/Michael

Top
#248752 - 23/06/2005 12:32 Re: empeg file structures [Re: mtempsch]
pgrzelak
carpal tunnel

Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
Do you mean this FAQ entry?
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs

Top
#248753 - 23/06/2005 12:38 Re: empeg file structures [Re: pgrzelak]
mtempsch
pooh-bah

Registered: 02/06/2000
Posts: 1996
Loc: Gothenburg, Sweden
Quote:
Do you mean this FAQ entry?


of course Don't see why my search didn't serve it up... possibly a speling error on my side.
_________________________
/Michael

Top
#248754 - 23/06/2005 16:12 Re: empeg file structures [Re: pgrzelak]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
Quote:
Do you mean this FAQ entry?


Well, sort of.

I build my database file directly, with my own code, so that's why there might be problems and I need to sort out the creation problem, rather than simply need the player to build one.

However, what I am now understanding is that when the player does build the database etc, it WILL save the files if the filesystem has been set rw, which answers my first question.

Top
#248755 - 26/06/2005 06:15 Re: empeg file structures [Re: ukengb]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
The problem with my created database file is to do with 'extended' characters. E.g. the length of the following track title:-

Cú Chullain

is counted by the empeg player (and by perl on my Mac) as 11, but when I create the database file on my RedHat linux server, it counts the length as 12. So whenever one of this type of 'accented' characters appears it gets the length wrong. I have tried to not use anything except regular ASCII, but some names/titles really need them and it's not practical to strip them out and have to monitor every new track, just in case.

So I really need to find a solution and count the length consistently.

I know I could let the empeg create the database each time, but this is something I specifically want to avoid, so it's not a viable solution. I just want to make perl on the RedHat machine count the same number of chars.

Anyone any suggestions as to how to do this?

Top
#248756 - 26/06/2005 06:41 Re: empeg file structures [Re: ukengb]
mtempsch
pooh-bah

Registered: 02/06/2000
Posts: 1996
Loc: Gothenburg, Sweden
Hmmm... what version of perl and how are you getting the length? If you have multiple accented characters, do you get 2 for each character or just a +1 error for the entire string?
_________________________
/Michael

Top
#248757 - 26/06/2005 07:37 Re: empeg file structures [Re: mtempsch]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
I get one extra count for each instance.

If I run this on the Mac (which counts the same as the empeg player)

perl -e '{print( "length = ", length "Cú Chullain","\n" );}'

it counts 11 chars, but the Intel RedHat machine counts 12.

If I modify that to start:-

perl -e '{use utf8; print....etc

Then RH perl counts 11 as I want, but the Mac gives an error:-

Malformed UTF-8 character (unexpected continuation byte 0x9c, with no preceding start byte)

Since I want this to be operable on both platforms, I still have a problem:-(

Top
#248758 - 26/06/2005 09:48 Re: empeg file structures [Re: ukengb]
mtempsch
pooh-bah

Registered: 02/06/2000
Posts: 1996
Loc: Gothenburg, Sweden
Feels like a version and/or latformspecific bug... i386 Debian with an ancient perl (5.005_03) returns 11 for the plain (non utf8) statement (don't have utf8 pm installed.)

Just toss another platform into the mix:
Activestate perl 5.8.7.build813 on XP returns 11 for the non-utf8 and 7!! for the use utf8 case (had to muck a bit with the quotes and parens to get it accepted, this is what I ran:
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "{print(length 'Cú Chullain');}"
11)
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'Cú Chullain');}"
7

Inserting extra [normal] chars in various places in the string do get noticed:
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'Cú1 1Chullain');}"
9
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'C1ú1 1Chullain');}"
10

But putting in the ú in various places it sometimes acts correct and adds 1, elsewhere I get a net negative change...and sometimes an error message. Can't see a pattern as to when/why it happens Some samples:
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'Cú Cúhullain');}"
8
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú Cúhullain');}"
5
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú Cúhuúllain');}"
6
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú Cúhuúllainú');}"
Malformed UTF-8 character (unexpected end of string) at -e line 1.
6
C:\Documents and Settings\tempsch.XP-TEMPSCH>perl -e "use utf8;{print(length 'úCú CúhuúllainúA');}"
Malformed UTF-8 character (unexpected end of string) at -e line 1.
6

Very funky...

I don't recall how to do it, but I think one can make perl check version and libraries version at compile time, but don't recall if you can do a conditional use or not... beeen a while...
_________________________
/Michael

Top
#248759 - 26/06/2005 16:22 Re: empeg file structures [Re: ukengb]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
My guess would be that the Mac uses one byte to hold the ú, but RedHat is using two. My point being that your source file has the characters encoded differently, with RedHat using UTF8 and the Mac using some 1-byte codepage.

To test, create a plain text file on both systems that both contain the ú. Then do an 'od -x' on both of them and see how the character is encoded. My guess, again, is that it'll be one byte on the Mac and two bytes on the RedHat machine.

If that's the case, then your solution is to programmatically figure out what encoding scheme is being used "natively" on each system the perl script is being run on. I don't know how to do that, though.
_________________________
Bitt Faulk

Top
#248760 - 26/06/2005 17:51 Re: empeg file structures [Re: wfaulk]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
I'm sure you're right, it's a variation in the encoding schemes of each machine. What I orignally used to encode the info for the database file is:-

$tagsdb .= pack("U2A*",$tags{ $_ }, length $$hot{ $_ }, $$hot{ $_ } );

$tagsdb is then added to the database file - for each fid.

Now I preprocess the tag data like this:-

my $utag = $$hot{ $_ };
utf8::decode($utag);
$tagsdb .= pack("U2A*",$tags{ $_ }, length $utag, $$hot{ $_ } );


This seems to produce the correct length count on both the Mac and RedHat and most importantly when I rsync the resultant database file the empeg player no longer tries to rebuild. So that appears to be the solution to that problem.

I am still having trouble with rsync though. After the first bulk upload rsync crashed with an error I wasn't able to record and the recent update also produced a different error when trying to turn swapoff after the sync:-

swapoff: /dev/hdc6: Cannot allocate memory

but the sync did complete, although I'm sure it transferred far more then it ought.

The beauty of rsync of course is that you can simply try again and it catches up from where it left off so you just keep going till it shows no files were tranferred and then you know it finished. I just need to keep doing it to iron out these last probs, but I feel that with memory so short it may be impossible to entirely eliminate all issues.

So, it now seems to count ok, but any advice on running rsync in low memory situations (or with no terminal) would be welcome:-)

Top
#248761 - 27/06/2005 11:46 Re: empeg file structures [Re: ukengb]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Quote:
...so the fact that some of the files are not there shouldn't cause a rebuild.


But it does -- the player checks that the count of FIDs on the disk matches the count in the database file. They don't match, so it'll rebuild.

As others have noted: the rebuild is in memory only. What's happening is that the player rebuilds the database in memory, and then attempts to write it to disk. Since the disk is read-only, it isn't written, and needs to be rebuilt next time. If (as the linked FAQ says), you have the disk mounted read-write when the player rebuilds the database, it will be written to disk, and won't need rebuilding next time.
_________________________
-- roger

Top
#248762 - 27/06/2005 13:09 Re: empeg file structures [Re: Roger]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14494
Loc: Canada
Does it really actually try the open()/write() calls in this case, Roger?

If so, then perhaps I should have a Hijack GUI entry to permit it to succeed.. or perhaps have it pop-up a yes/no prompt.

??

Top
#248763 - 27/06/2005 14:30 Re: empeg file structures [Re: mlord]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Quote:
Does it really actually try the open()/write() calls in this case, Roger?


Yes, I believe that it does -- if open() fails for write, it just skips over the write(). Of course, it's been a long time since I looked at the code, so I could be mistaken.

Something in Hijack that allows it to be written would be cool -- presumably it'd remount rw underneath the player at this point?
_________________________
-- roger

Top
#248764 - 27/06/2005 14:39 Re: empeg file structures [Re: Roger]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14494
Loc: Canada
Yes, that's the idea.. remount r/w during the call to "open()", and then remount r/o again on "close()".

Cheers

Top
#248765 - 27/06/2005 19:16 Re: empeg file structures [Re: mlord]
Mataglap
enthusiast

Registered: 11/06/2003
Posts: 384
Quote:
Yes, that's the idea.. remount r/w during the call to "open()", and then remount r/o again on "close()".


That would be way cool. Thanks Marki!

--Nathan

Top
#248766 - 28/06/2005 06:34 Re: empeg file structures [Re: mlord]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Quote:
Yes, that's the idea.. remount r/w during the call to "open()", and then remount r/o again on "close()".


Sounds good -- bear in mind (and this is fairly obvious, so my apologies if you've already thought about it) that the player's writing 3 files (tags/playlists/database[3])at this point, so if you do prompt yes/no, it would be a good idea to only do it the once, lest user error cause the player to only write half of the needed information.

Unfortunately, I don't know the order that it writes them in.
_________________________
-- roger

Top
Page 4 of 4 < 1 2 3 4