Greetings!

Just checking to see if anyone else has had similar trouble...

I have recently upgraded my players to 100GB drives and version 2.01 and hijack 413. Sadly I did all three at the same time, making troubleshooting more challenging...

I am now seeing what I can only describe as a caching glitch. The start of each track now has a noticeable audio hiccup at approximately 17 seconds into the track. It is a short glitch, I suspect as the drives are read for more data. The effect is more obvious / blatent if I am skipping between tracks or if I perform other disk writes (setting bookmarks, etc.).

Now, the extra memory has been in place and working fine for a while under 2.0 and hijack 398 (my previously loaded version). So I do not suspect the memory enhancement.

This happens both in car and on AC power (docking station), with autodetected power or hijack forced AC/DC settings. There are no console error messages via hyperterm when connected, so I do not think it is a disk error or IDE issue. In fact, there are no spinup or informational messages at all - was that removed from 2.01?

I am currently building another empeg into the same combination of software / hardware to verify this. When that is complete (likely this weekend), I will try to duplicate this on another player. Unfortunately, I will also not be able to try any downgrade testing of the player software or hijack until that build completes. I am in the middle of a four day rsync...

Hardware: empeg Mark2A, 48MB RAM, dual 100GB
Software: developer 2.01, hijack 413, no other software, no cache statement in config.ini
Bootlog: memory / caching statements...
Memory: 47452k/48M available (984k code, 20k reserved, 692k data, 4k init)
Using non-standard cache size 614 (bonus 32Mb, adjustment 0)

Oh, and if it is of interest, I am one of those people who can no longer reboot the player through hijack or when doing an upgrade or kernel flash through logoedit. Any of these automatic reboots will fail and require a hard reset. This is for all of my players, so I suspect that the memory modifications have changed something there. Different item I know, but as long as I am posting...

Anything like this?

I never did figure out what the problem was with mine. It was my Mk2 backup. Once I got my Mk2a back from Mark, I swapped the drives in and the problem was gone. Maybe it was something on the Mk2 board (ide) or maybe it was the lower amount of RAM (more likely). My problem was likely do to not enough RAM with my database size to allow for proper cacheing, yours (like you suspect) may be due to a bug that doesn't get full use of your RAM.

Didn't Mark post that 2.01 had trouble detecting added RAM in some situations? I think he added a workaround in HiJack to address this.

Your reboot problem is odd...

Greetings!

Similar. Because I was messing with the drives and player versions, I am not certain of the cause. The symptoms are very similar though. I would have expected an error to appear if it was a drive related issue (bad track, block, error, timeout, etc.). Still, I will investigate further this weekend.

The reboot issue is a known problem. Something is not being cleared or is not initialized when a reboot is done. Mark noticed it as part of the hijack reboot, and noted that some players showed this behavior while others did not. But this is an added data point - it is not just hijack - any reboot will cause the same symptoms.

My upgraded player has "only" 32MB total RAM, and does not exhibit this problem (audio glitch) for me with either v2.01 or v3alpha8 and the latest Hijack.

So.. I wonder if it's a quirk that is amplified by having even more memory.. 48MB (or 64MB) ?

Could someone else with a 48MB+ player please try to reproduce this issue.

Thanks

PS: My 32MB unit also fails to reboot ("data abort vector..") nicely these days, though my 16MB unit is fine. Go figure.

Cheers

Oh, here's an idea to try, Paul:

Modify Hijack's auto-spindown feature for longer timeouts:
Code:

---

[hijack]
spindown_seconds=60

---

The theory here being that with your HUGE memory size, it's possible that the read-ahead is fetching 30 seconds or so of data off one drive, wherein the other one spins down, and then the spun-down drive is accessed and a pause occurs as it spins up again. The default timeout is 30 seconds; increasing it to 60 should eliminate any such worries.

Cheers

Hmm... No luck. The first song I tried seemed better, but after skipping around a bit I noticed the same glitching.

I haven't noticed any audio glitches with 64MB. I have noticed that the visuals get sluggy whilst the huge cache fills though.

I wonder whether the increased kernel size on big memory players has anything to do with this, and whether a Reservecache line would help?

I am doing a bit of experimenting now. I am also turning on the disk status icon, so I can see what the drive status is. I normally have it turned off... More to follow.

Interesting. It looks like the drives are always spun up (the hijack option set to 60 seconds), and that it did seem to help a bit. When I heard the audio glitch, it seemed a bit improved. There also was the W icon version, so the audio thread needed input. Now, even without glitching I would see that W appear, but only very briefly and without any glitch. During a glitch, the W stays on for a longer period of time (the length of the glitch). Still no drive errors or anything - I will try the reservecache option next.

And we have a winner!!! The addition of the ReserveCache option (set to the default of 180) completely eliminated both the quick Ws and the long Ws / audio glitches. Thanks!!! Now, why would the absense of this value cause problems...???

config.ini (before and after with [labels])
empeg:/empeg/bin# diff /drive0/install/config.ini /drive0/var/config.ini
[Startup]
> ReserveCache=180
[display]
< caching=0
> caching=1
[hijack]
> spindown_seconds=60

boot log fragments (after - see earlier in the thread for before):

Memory: 47452k/48M available (984k code, 20k reserved, 692k data, 4k init)
Using non-standard cache size 434 (bonus 32Mb, adjustment 180)

Edit: As a final test, I reverted all other changes out (drive status display and spindown timer). The player is still working perfectly! There is something about not having that ReserveCache line that is triggering glitching... Interesting!

Yeah, this is because the kernel has to allocate extra memory to keep track of all its extra memory! The 'data size' reported by big_mem players differs greatly from that of stock due to this.

Basically the player application makes an assumption of the kernel's size when calculating the cache size. Mark and I have had a few discussions on how to deal with this such that ReserveCache wouldn't be needed purely to deal with an increased kernel size (thus allowing more/bigger hijack features), but unfortunately the player's calculation borks on 16MB players if hijack misreports the memory size to achieve this. (Mark tried this in a series of kernels around v400)

For the moment the issue is simmering on the back burner and ReserveCache is the recommended way of dealing with increased kernel size. If the player application gets modified to allow hijack to deal with this more gracefully, or if Mark or I think of another way to do it then that requirement could go away.

180 sounds like a lot though. My quick guesstimate suggests that ReserveCache=6 should be enough to cover the additional 324k of data space. Can you verify this? (If it isn't then try 8,10...)

Greetings!

No worries. This is a simple fix, and I will test adjusting the size of the ReserveCache option when I return to the office tomorrow morning. I am currently not able to get a console (or other) connection to the player at the moment due to insufficient connections...

I made the mistake of typo'ing 160 when I meant 16. My player just kept starting and crashing. Blah.

And I see that my wife never sent out the box with the empeg motherboard and memory to mlord, either.

Greetings!

Testing. But I don't think you will like the results...

Yesterday, I did not notice any problems with the ReserveCache=180. Today, I tried lowering that value to more reasonable numbers. Suffice it to say, I am leaving it at 180. I started to notice increasing instances of the drive icon showing the "W" (audio thread waiting for data) at around 160, and started hearing glitches around 145.

When I saw the glitching (Ws every second or so, with a big audio glitch around 17 seconds) at 6, 8, 10, 12, I decided to do the binary search approach. This brought me to a breakdown point of around 145-150...

Please note, I am not your typical user. My database is unusually large, and my FID count is off the map. So sane computations may not help. Of course, all of this is complicated by VBR files - I could tell that some files were more "trouble" than others, based on the bitrate and the amount of data needed / rate of cache drain. Also, I noticed that the glitches seemed to be made worse (to the ear) by having auto volume adjust in place - perhaps the wait / glitch in the audio thread is being interpreted as valid audio by the algorithm.

For me, a value of 180 for ReserveCache (a level where I only see an occasional W for very complex tracks) is an acceptable tradeoff, but I will still try downgrading to 2.00 this weekend.

Testing methodology:

(previous test complete)
q
rw
rwm
vi /drive0/var/config.ini
(change setting)
rom
ro
sync
hard reboot player (remove from sled)
start player playing
skip ahead at least six tracks (to make sure nothing cached)
play for at least 30 seconds, noting any Ws or glitches
repeat.

Anything audible failed, more than 3 or 4 Ws (depending on the complexity / bitrate of the track) was considered marginal. At 160 and below, I started seeing more Ws but no audio glitches. 150 brought glitches to more complex songs, 145 glitches in more songs than not.

SW: Player 2.01 and hijack 413 only. Auto Volume Adjust (low)
Highest FID: 0x7E4D (32,333)
empeg:/empeg/bin# ls -al /drive0/var
total 4484
drwxrwxr-x 2 root root 4096 Aug 27 15:55 .
drwxr-xr-x 6 root root 4096 Aug 27 15:34 ..
-rw-r--r-- 1 root root 933 Sep 3 05:10 config.ini
-rw-rw-r-- 1 root root 4415935 Aug 27 15:54 database
-rw-rw-r-- 1 root root 137480 Aug 27 15:55 playlists
-rw-rw-r-- 1 root root 376 Aug 27 15:54 tags

As near as I can guess from observing the player software, each ReserveCache increment (in v3alpha8 at least) corresponds to 64kB of memory. So a ReserveCache=180 setting means that Paul is "throwing away" about 11MB of his new memory, bringing the 48MB player down to 37MB or so.

Seems a tad excessive. Linux memory management overhead is about 5% or so on most kernels, so I would expect a setting of ReserveCache=32 to be more than good enough.

If you could give that value a try, it might help us understand it better.

EDIT: I suppose we should also allow for Pauls BIG database here, another 5MB or so, giving a total of ReserveCache=112 for his player.

Also, with current Hijack versions, there really should be no opportunity for the disk I/O to starve the audio playback threads (important bits of this code are attached here for scrutiny), so I don't think it's the reading ahead that should be the culprit here.

EDIT: Ah, nevermind.. I see that Paul has already tried values between this and 180 without success

Cheers

Peter, how large are the chunks of memory which correspond to increments of the ReserveCache parameter? My observations suggest to me that each increment is 64kB, but it would be quite useful here to know for sure.

Thanks

Ooooh... I just got it to happen here on my 32MB player with v2.01. If it's repeatable now, I oughta be able to fix it..

-ml

If this is of any help, one observation is that the glitching seems to occur (first and worst instance) at about 17 seconds. In cases where I see this with the drive status displayed, I can see the audio thread waiting (W icon) well before the 17 second mark, in very short bursts. It is only at that 17 second point (plus or minus 1 second) that there is a big audio thread starve (W icon that stays on for more than a few frames - 1/75ths of a second). This resolves, but there is a blatent audio glitch and it seems to knock the auto voladjust off (perceived sound level changes and fluctuations) for a second or so. Then it smooths out.

I will get repeated instances later in the song, but usually not as bad as that first hit.

Cool! Repeatable is good!!!

Okay, ReserveCache fixes all but the very first glitch for me, even if I completely comment-out the Hijack CPU-yielding hack.

Based on various observations (a bit too complex to explain here), what is happening on my player is a very obvious "low memory" condition, which is causing various threads to block on memory allocation when they normally would never have to block.

This seems to only happen when the disk read-ahead in the player is busy, which makes sense since that activity will tie up a lot of pages in the kernel for holding the disk reads before they get copied to userspace, making even the most trivial of memory allocations result in a process blocking.

For example, opening an FTP connection to my in-kernel kftpd was NOT POSSIBLE during the disk I/O, due to lack of memory for the new thread's stack allocation (it needs 12kB total: 8 for stack, plus 4 for a parameter/buffer block). I saw other indicators of extreme low memory as well.

Pity we don't have O_DIRECT in this kernel version --> O_DIRECT forces disk reads directly into the userspace buffers, bypassing the page cache completely for apps which roll their own, like the player.

As a sidebar, I recently added the "--direct" flag to my hdparm utility, to permit benchmarking with/without direct userspace I/O, and it the difference with the RAID controller I'm working on is most impressive: a 4-drive RAID10 jumps from 70MB/sec to 130MB/sec, and a 4-drive RAID0 whooshes along at 206MB/sec. On a slow CPU using 64-bit/66-Mhz PCI, that is.

Now, for the first glitch, which still can happen. I don't hear it, but I see the W icon flash, which means the software knows it had a glitch. The Hijack CPU-yield on disk I/O is imperfect --> it misses the first FID that is read, so it is possible for a large initial track that the readahead of that same track may still glitch, on both v2.03 and v3alpha. I'm unlikely to fix that one.

Cheers

Okay. I follow. Basically the cache is so large that the call to fill it starves out memory requests from pretty much everything else. Especially with an extra large cache, because the kernel memory used for the initial read / processing takes lots of memory away from other kernel processes. Shrinking the cache makes the problem less of an issue because there is less to fill and the transfer happens quickly enough and are small enough that the other processes are less likely to get choked out by it.

Nice RAID transfer rate jump, by the way!!! Significantly greater throughput!!! COOL!

Edit: And since it is totally dependent on memory size and kernel, the upgrade to 100GB drives is not the issue, other than the side effect of a larger database. And it was not an issue previously with the 80s because the database size was smaller and I was running under 2.00 / older hijack. The only other test I would be curious about is if this would also happen under 2.00 - to see if the memory enhancements made there changed matters or if it is an inherent problem.

Paul, is there any update on this?

I'm running 3a10 with a 64mb memory card and I'm getting more memory related errors than I had before with 2.00final and no extra RAM. I'm needing to play with ReserveCache settings that I never needed before. I'm wondering what benefit I'm getting from the memory card upgrade.

For instance, switching tracks on the player does not display a new "Vcb: 0x428fa000" via serial. If the player is running, I can't connect via emplode. When the player boots up, it usually boots to audio but with no display (LED is still pulsing, blank screen). Usually turning the player ap off makes things like ftp work (rather than time out). After a boot, the menus crawl very very slowly. I never had any of these issues before. Is this stuff that I'll get with 2.01 or are these alpha issues?

I'm now at a ReserveCache of 180, up from 50, up from no setting at all. These changes only helped me from getting:

VM: do_try_to_free_pages failed for kswapd...
VM: do_try_to_free_pages failed for kswapd..
no room for private writable mapping
error: -12

It just doesn't seem like the alphas or 2.01 work too well with extra RAM? Or do I just need to tweak this stuff more? I was going to switch to 2.01 and give up Crossfading, but after reading your post, maybe I shouldn't expect too much of a difference.

Well, if I remember correctly the 3.0 alphas have a larger memory footprint than the 2.0 variations. I have not tried the alphas yet, so I cannot give you a comparison point.

I am running 2.01, I have my ReserveCache set to 180, and that seems to work for me. I am able to console into the player and ftp without a problem. I have never seen the slow menu problem, personally, so I suspect it is alpha related.

The easiest test is to drop to 2.0, delete and rebuild your databases. Because this version is not aware of any extra memory, you eliminate that from the equation. You would still see some benefit at the kernel level, because hijack will still access it, but the player application will not try to grab from the extra RAM for its cache. It is a good check to see if the behavior changes.

If this works, then move up to 2.01 and compare the differences. If you are running more background applications than I am (I just run hijack normally) you may need to set aside more memory for those.

By the way, I am sorry to read in the other thread that your empeg is smoking again.

My player should be back to me this week.. I currently have my drives in a backup player and am thinking of installing 2.01 on the drives so that when the player arrives, I can just swap the drives. Is there a problem running 2.01 on a player with only 16mb? Is the cache sequence too aggressive for 16mb?

I'd like to get it running stable on 2.01 before I switch to 3.0. Also, I doubt my player will handle 3.0 with 16mb and the size of my database. My drives probably won't see any 3.0 variant until I know my hardware issues are behind me and the backup is on the shelf.

Paul, I'm running 2.01 now with the memory card. With no ReserveCache setting, I was getting a lot of low memory errors over the serial output. I then tried 180 and the errors went away but I was still getting an interupted audio signal at the beginning of some tracks. Lots of fast W's too. Then I switched to a setting of 200 and the audio is better (bur still stutters sometimes). The W's are still coming (quick ones) but not as often.

What's odd however is that emplode can not find the player at all unless I turn the player off first... have you had this happen? If I keep bumping the ReserveCache up, might it fix it? 200 seems high already...

EDIT: I can't access the player via html interface, emplode (even to simply add songs to the queue, or ftp) if the player AP is running... I'm upto 220 ReserveCache now. This is going to make DJing impossible if I can't add songs to the playing order remotely.

Hmmm... How much extra memory did you add? I am at 48MB. Unfortunately, I suspect you may have to downgrade to 2.0 if the reservecache option becomes too high or impractical for you. Worst case, with 2.0, the memory is only availble to the kernel because the player application is not aware of it.

I have not had trouble with emplode not detecting the player, but if there is enough memory starvation, I can see where it would happen. Try bumping a little higher and see if that helps you. If it doesn't downgrade to 2.0 and use the RAM for additional background kernel apps.

I just don't get it... I have more RAM than you (64) and you have more FIDS but I get quick W's and can't connect to the player if the ap is running while you don't have these issues...

I just did a ReserveCache of 280 (!) and emplode found the player between songs... But now it's hanging on getting the database...

Upto 320 and still getting the same results.... empegface has a framerate of about 1 frame every 6 seconds IF it sees anything at all..

I thought these 64mb cards were supposed to be the cure for all this crap... useless at this point and not worth the hundreds of dollars I spent to order one and get it installed. I no longer get memory errors on my serial output, but now I have new problems that effect my ability to use the unit. If I never had a serial connection, I never would have noticed the errors before. No stuttering audio and no W's.

LOL, now this is almost getting funny... my player is doing this after about 10 minutes of play:

Quote:
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32
ksock_rw(send): error: -32

It is repeating over and over.

ksock_rw is part of the Hijack FTP/HTTP daemon.

That's the problem - you have more RAM. The caching routine is designed around the normal 16MB. It is extremely aggressive, in that it completely tries to fill the cache each time. The issue is that it performs the disk reads and buffers the data into the kernel memory space, flooding out other processes. While memory support was added to the 2.01 kernel, there were no known changes to the caching mechanism. Thus the aggressive caching tuned to 16MB becomes a psychotic serial killer with 64MB.

The error you are seeing might be another symptom of this memory limitation. Question - do you notice the same behavior if the player is paused? This is just a test to see if it is the cache (assuming that the paused player fills the cache and then rests once it is full) or the player application in general.

How about if I have an even larger ReserveCache and only leave the player about 16mb or so?

Putting the playing into Pause resolves the issue. Emplode finds it just fine. Of course, I can only put the play into Pause via the player itself because hitting the pause button in emplode or empegface doesn't work.

So am I basically out of luck until a software release takes advantage of the extra memory?

The W's indicate that a thread is waiting for disk. It implies you either have a remarkably slow disk (like the original Toshibas shipped in 1999) or that the disk is generating a lot of errors. It has very little to do with the amount of memory you have installed, or the amount you are reserving for cache.

If you get audio stutters, the most likely place for this to happen is near the start of a track, within the first 5 seconds of playback. The playback cache spits out what it has cached and is expecting more data from the disk, which it is not getting in time - hence a brief pause. No amount of reserved memory will help in this situation if the data is simply not being delivered on time.

Other, continuous audio stutters occuring rapidly during playback also point to a failing disk or one with high latency.

ksock is a kernel socket operation. The socket error -32 is FFE0, the meaning of which as an Errno in Unix escapes me at the moment.

This implies you have an IP stack fault of some sort. Have you had serious electrical problems on this unit? It may imply subtle electronic damage which may also account for the apparent slowness of the disk, as the ethernet chip and the IDE interface are on the same buffered databus/5V power rail. You may have a damaged bi-directional databus buffer chip which is causing data corruption.

Well crap.

Things are fine with 2.00 except for serial output giving me memory errors... Maybe I can fence some stuff on eBay and afford to mail it to you... If only the damn Euro would fall.

What kind of memory errors with 2.00???

Quote:
If only the damn Euro would fall

Well, you'd need to talk to Mr. Bush about that I'm afraid....

Isn't this just part of the notify option from config.ini? Or perhaps some debugging data? It does not seem like an error.

Well, I still think the memory upgrade is a good thing. It is great if you use third party applications, or if you use the HTML interface a lot. I have a bunch of people at the office streaming from my player at the same time, and they do not seem to be stepping on each other. It is also great for big fsck runs...

But, yes, the player application itself does not make good use of the memory yet. At least not the 2.0x class players. I have not tried the alphas yet.

I know what you mean about the LEDs - I have one I need to get fixed as well. The more modifications, certainly the more things can (and usually do) go wrong. But I still think that it is worth it.

And, yes, that button hack is still extremely impressive!!!