Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Topic Options
#307747 - 28/02/2008 21:07 3040.570 NOMEM ERR
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
I'm in the process of doing a massive file download (>100 GB) from my empeg to my desktop computer, creating a backup.

While doing the download, I am also listening to the player.

During this, I have gotten several 3040.570 NOMEM ERR on the screen of the empeg. When that happened, the empeg continued to play until reaching the end of the play-ahead buffer and the file downloads apparently continued, but the player controls were unresponsive. When the track currently playing finished, or possibly the next file for download was reached, or possibly when the next song to play was reached (don't know which of these three things might have been the trigger) the error message went away and things continued normally.

Is this behavior caused by my overloading the player with simultaneous playing and downloading, or is this the harbinger of coming trouble?

tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#307749 - 29/02/2008 02:07 Re: 3040.570 NOMEM ERR [Re: tanstaafl.]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Buggy software -- something somewhere is not checking/handling a NULL return value when allocating memory on the fly.

Ignoring that problem, your downloads will take less than half the time if you PAUSE the player while they're going on.

Eg. only 2 days instead of 4 days for a full dump -- that kind of difference.

Cheers

Top
#307770 - 29/02/2008 19:06 Re: 3040.570 NOMEM ERR [Re: tanstaafl.]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
Quote:
Ignoring that problem, your downloads will take less than half the time if you PAUSE the player while they're going on.


True, but, surprisingly time is not an issue here. It is sitting on my desk at work entertaining me while it merrily chugs away at replacing the backups lost when a RAID repair went awry. I listen to it during the day, then turn it loose to run full bore overnight, and if it takes several extra days, no problem! smile

tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#309615 - 29/04/2008 23:27 Re: 3040.570 NOMEM ERR [Re: tanstaafl.]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Hi,

I have seen the a similar nomem err with downloading a relatively small database <6 GB. I am using the new v10/v9 build and player software.

The error is "0e80:0 nomem err" or sometimes "0000.0 nomem Err and "0000.1 nomem err"".
The front panel locks up but you can use the remote to get arounsd it sometimes although, it returns quickly.

I have not seen this before on this player with the multiple times I have loaded the system while testing the v10/v9 build/load (probably 30 times) and previously loaded up 45 GB to the drives.

I don't think I have seen it until I started using the Ethernet load method instead of the USB load. I am wondering if it is related to the build or Ethernet.

It would surprise me if it was Ethernet...

I will try a load with USB. Check it out, see if it works.

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#309622 - 30/04/2008 10:16 Re: 3040.570 NOMEM ERR [Re: Ross Wellington]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: Ross Wellington
Hi,

I have seen the a similar nomem err with downloading a relatively small database <6 GB. I am using the new v10/v9 build and player software.

The error is "0e80:0 nomem err" or sometimes "0000.0 nomem Err and "0000.1 nomem err"".
The front panel locks up but you can use the remote to get arounsd it sometimes although, it returns quickly.

I have not seen this before on this player with the multiple times I have loaded the system while testing the v10/v9 build/load (probably 30 times) and previously loaded up 45 GB to the drives.


Is this with two 250GB drives installed ?

I think it is very likely that there is simply insufficient RAM in the empeg to hold of the necessary data structures (kernel and player) for filesystems of that size.

You may be able to work around it, by shifting more memory away from the player software, using the ReserveCache=nn flag in the player's config.ini file:


[Startup]
;;
Tell the player to leave about 1MB of RAM for other uses:
ReserveCache=15


Oddly, the Empeg FAQ entry for this suggests that each chunk is reported to be a bit larger than 32kBytes -- we've known for a few years now that each chunk is actually 64kBytes.

Cheers

Top
#309623 - 30/04/2008 10:30 Re: 3040.570 NOMEM ERR [Re: mlord]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
Originally Posted By: mlord
Oddly, the Empeg FAQ entry for this suggests that each chunk is reported to be a bit larger than 32kBytes -- we've known for a few years now that each chunk is actually 64kBytes.

The original releases did use 32K chunks, but I can't remember when we changed it to be 64K... possibly it was as long ago as between v1 and v2, which would indeed mean that most people on the BBS are probably using 64K now.

Peter

Top
#309632 - 30/04/2008 14:01 Re: 3040.570 NOMEM ERR [Re: peter]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31600
Loc: Seattle, WA
Updated.
_________________________
Tony Fabris

Top
#309651 - 30/04/2008 23:21 Re: 3040.570 NOMEM ERR [Re: tfabris]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Hi,

Yes, it is 2 each 250 GB drives.

I need to try the config.ini modification.

I will let you know.

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#309731 - 03/05/2008 06:29 Re: 3040.570 NOMEM ERR [Re: Ross Wellington]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Hi,

The config.ini change did improve the situation. It used to come up immediately and then a few seconds to a minute later. The display is as "hung" and you can still use the remote.

Is the problem related to the fact that I have 2, 250 GB drives and managing them together or is it related to just drive size in GB and managing a larger drive?

Is there another thing I need to try? I could try different sizes of cache, I guess.

I have an additional RAM kit that I could install if it is a more RAM situation.

Thanks in advance,

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#309790 - 04/05/2008 11:46 Re: 3040.570 NOMEM ERR [Re: Ross Wellington]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
I think the problem is very likely that there is simply not enough utility RAM for the kernel to keep track of the filesystem data structures for two very large filesystems as you have there.

Increasing the ReserveCache= setting should help a lot -- every count of 16 represents a megabyte, but then that memory is taken away from the player's own cache (and thus the root of the problem: player s/w trying to maintain a duplicate of the kernel's own cache, thus taking near 2X the RAM in doing so).

So, try perhaps ReserveCache=48 -- I don't know how much RAM it really needs, but that ought to quiet things down a fair bit. If it does, then installing the RAM upgrade kit will help a lot as a longer term solution.

Cheers

Top
#309798 - 04/05/2008 12:23 Re: 3040.570 NOMEM ERR [Re: mlord]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
Originally Posted By: mlord
(and thus the root of the problem: player s/w trying to maintain a duplicate of the kernel's own cache, thus taking near 2X the RAM in doing so)

I'm not sure what you mean by this. There certainly isn't enough RAM in the player for the kernel's block cache to get to anything like the same size as the player's chunk cache. The player uses mlock, so it'd be more accurate to say that the player "successfully overrides" the kernel's cache, rather than "tries to maintain a duplicate". The extra memory used (as compared to, say, using O_DIRECT; mmap was buggy in ARM kernels that old) is the size of a single chunk: 64K. Which the kernel then evicts from (what it finds to be) its very meagre block cache when then next 64K is requested.

Peter

Top
#309807 - 04/05/2008 17:04 Re: 3040.570 NOMEM ERR [Re: peter]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Originally Posted By: peter
The player uses mlock, so it'd be more accurate to say that the player "successfully overrides" the kernel's cache, rather than "tries to maintain a duplicate".


The player's chunk cache is also used for "intelligent" look-ahead based on the current running order, which (AFAIK) we couldn't have relied on the kernel's block cache for.
_________________________
-- roger

Top
#310045 - 10/05/2008 23:48 Re: 3040.570 NOMEM ERR [Re: Roger]
altman
carpal tunnel

Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
...plus the whole problem with a thread unexpectedly blocking because it tries to access some memory that the kernel has evited from the cache...

Top