So I discovered that the Ogg encoder I had used to encode my test song was 1.0RC2. According to the Tremor mailing list, songs encoded with older versions take more CPU power to decode. So I tried re-encoding my test song with the 1.0 final release, I was able to to play it back without gaps as long as the quality setting was set to 7 or less. At 7, it's right on the edge, meaning scrolling through menus or other activity will often make it skip.
Anyway I'll be packaging this up into a receiver.arf file soon. If anyone with a larger Ogg collection tries it I'd like to know how it works. Also, what quality setting do you Ogg users usually use to encode with?