It's not closed at all - yes, with the "consumer" image on the player, there's just the player and nothing else on the hard disk. However, if you use the "developer" image, you can quit the player and get a shell prompt. On the disk are the usual unix tools, glibc2.1, plus rz/sz (download your own programs to the unit). /dev/hda2 is a 32Mb blank (not mkfs'ed) partition for your own stuff which won't get blatted by upgrades.
As for the protocol the player talks when running, we'll shortly be publishing the source to a client which talks to the server running in the player (read/write files, control the player, grab screen images, etc). Remember our programming effort here is limited, and it's currently best spent improving the software on the unit rather than writing docs on the internal protocols for people to read :)
Hugo