This is mostly aimed at Mark Lord, but this aint a BBS for nothing!

Great work on the Hijack kernel! It rocks! It also leads me to believe that you know a little something about Linux, and since you supplied the "precompiled toolchain", if you don't mind too much I have a few questions...

First, a little about where who I am so you can gauge your answers: I have been developing software on the PC platform for some years on the MS side. I have written everything from DOS and Windows user apps to device drivers for the WIN 98/NT/2000 platform, I even wrote a custom multitasking kernel for the PC platform before windows was usable. I design custom embedded hardware, distributed systems, cards for the PC bus, etc. I write firmware in C and native assembly for various microprocessors, microcontrollers, and various DSPs. I believe that Linux is "the wave of the, well, now and future" in terms of embedded devices. I actually originally bought my empeg because it was cheaper than the StrongARM development kit (not to mention much cooler and more useful). So, now I am beginning to launch into Linux, but I have a few questions . . .

I have RedHat 7.2 installed on my PC. It works great.

I have downloaded your "precompiled toolchain" and unpacked it.

Q1: What is the deal with this ".tar.gz" thing. Why is everything Linux packed down with two different tools? I have been using PKZIP and later WinZIP on the PC side for years, as well as others (LZH, etc), and every one will handle the directory structure as well as the compression.

Q2: When I unpacked the "precompiled toolchain", it expanded into "/usr/local/armtools-empeg" and the readme says:

export PATH="/usr/local/armtools-empeg/bin:$PATH"

But be careful: don't include this on your PATH when building non-Empeg stuff,
because the cc1 executable will get confused.

but all of the tools in the bin directory are named like "arm-empeg-linux-gcc". However there is a directory in there that looks like "/usr/local/armtools-empeg/arm-empeg-linux" under which there is another "bin" directory with executables like "gcc". I got nowhere with

export PATH="/usr/local/armtools-empeg/bin:$PATH"

but

export PATH="/usr/local/armtools-empeg/arm-empeg-linux/bin:$PATH"

seems to work better. So the question is what's up here? I must be conceptually missing something here. And what is cc1? (Maybe that's what I'm missing?)

Q3: I found a couple of example apps on the "Unofficial developers site" (www.empeg.mars.org). But I could not get a good compile. Specifically I tried pngtest-0.1-empeg-arm-linux.tar.gz

It says that you need to obtain "libpng" and "zlib". I found those, downloaded them, and after pointing my path to "/usr/local/armtools-empeg/arm-empeg-linux/bin" I was able to build both of those libraries apparently with the arm gcc (as the build was successful, but the x86 test app stopped working) but I still get a truck load of errors when attempting to build the "pngtest" example.
So, the question is, can you provide a simple sample app that can be built with the cross compiler so that I can tell that I have it installed and working properly (or al least point me to one)? Is there already something of that nature in the "precompiled toolchain" that I am just missing?


Thanks,

Steve

Q1: What is the deal with this ".tar.gz" thing. Why is everything Linux packed down with two different tools? I have been using PKZIP and later WinZIP on the PC side for years, as well as others (LZH, etc), and every one will handle the directory structure as well as the compression.


Linux and the GNU toolchain follow the UNIX philosohpy that a toolbox containing many simple tools that each do one thing well is better than one tool that does everything but no so well.

In this case, the 2 tools are gzip which is the compression tool, and tar which is an archive tool. (tar is shorthand for Tape ARchive).

This combined with the power of various shells allows very powerful things to be done, e.g.

(cd /mnt1 && tar cpXz .) | ( cd /mnt2 && tar xpzf -)

which will copy the contents of one mounted drive /mnt1 to a second mounted drive /mnt2, preserving file structure, ownership and permissions along the way. Very useful during a drive change

You will find this philosophy all over UNIX and linux. Ironically, tar does understand compression. The 'z' switch to tar will tell it to zip/unzip the file as appropiate. So tar cpfz mydirectory > mydirectory.tgz actually creates a zipped archive of the directory. (.tgz files are the same as .tar.gz files to all intents and purposes)

Try typing 'man tar' and 'man gzip' in a shell.

(cd /mnt1 && tar cpXz .) | ( cd /mnt2 && tar xpzf -) 

Well, why exactly you using the "z" command line option there? It only slows down the copy in that case.
It would be useful however if you did something like

(cd /mnt1 && tar cpzf - .) | (rsh somehost '(cd /mnt2 && tar -xvpzf -)')

At least if the link between your computer and somehost is slower than FastEthernet. Anyway, back to the question you answered:

There are two possible answers actually. The first (and more common one) is the KISS principle that you explained. (KISS: "Keep It Simple, Stupid" or "Keep It Small & Simple")
The second one is the historical approach: The first tools on unix where small and simple, and the system evolved from there. So instead of hacking some additional code into an existing tool, unix programmers just created an additional tool that did just that one additional task. And this proved to be good decision. this way, the same archiving tool (tar) could be used with many different compression tools (compress .Z, gzip .gz, bzip .bz and finally bzip2 .bz2). But also, the same compression tool (gzip for example) could be used on bare files or with different archiving tools: ar, tar, shar ...
So the question remains: Where the tools small and simple, and followed the KISS principle from the very beginning, or where they small and simple and that later proved to be a good concept?

On a different question:
cc1 is one of the executables gcc uses during the compile process (along with the C preprocessor cpp and others).

cu,
sven

...or if you have remote mounts. But, yes you are correct...that's what happens when I hastily type a reply

Q1)
One thing that a lot of people don't realise though with .tar.gz or .tgz files is that you generally get better compression - particularly when you have alot of little files. tar joins them all together into one big file and gzip compresses them. bzip2 is also coming into use a bit now and seems to provide even better compression again.

Q2)
This kinda confused me too, but since it was not hard at all to get working I didn't investigate it any further. Just do exactly what the readme says and you will compile kernels fine.

When it comes to custom apps I simply compile with arm-empeg-linux-gcc which is in the PATH the README tells you to set up. I'm not sure about that second path you have found.

{soapbox}

OK, OK so Q1 was sort of my little jab at Linux. I already knew that the basic answer is that it is just a legacy issue, and that there are, in fact, many tools that will archive AND compress (i.e. even old TAR with the -z parameter). I also know that most Linux folks believe small, single purpose apps stacked up to provide larger overall functionality to be a good approach.

Since 99.9% of EVERYTHING Linux is .tar.gz (or more recently .tar.bz2) and the vast majority of the time I just want to download and unpack things, it only makes sense to me to simply write a script to handle it. So, my very first Linux script: "untargz" goes something like this:

#!/bin/bash
gzip -d $1.tar.gz
tar -xvf $1.tar

there is probably a better way, but this made simple (KISS) sense to me.

Now where

(cd /mnt1 && tar cpzf - .) | (rsh somehost '(cd /mnt2 && tar -xvpzf -)')

may be a very clever trick, I think it could be argued that this is far from the KISS principal. Something that would be much simpler and make more sense to me would be something like

copy /mnt1 somehost/mnt2 --preserveall

but maybe I am just too simple minded.

Somehow the rather cryptic 'tar' trick reminds me of the clever fellow who wrote an entire multitasking OS in one line of recursive C code (100,000 chars long and impossible to read or debug).

I guess that this was just my frustrated outlashing about the cryptic nature of the Linux shell language in general. I find myself constantly frustrated at things like:

ls = get a directory LiSting
su = create a shell with a new user (SuperUser)
rsh = run a command on a remote host (RemoteSHell)
rm = delete file (ReMove)
rcp = copy a file between two machines (RemoteCoPy)
cp = CoPy a file

I could go on, but the point I am trying to make is that it has never stricken me a simple to take two letters out of a command like COPY and turn it into CP, or seven letters out of SUPERUSER and turn it into SU. Now, these are simple cases that are fairly easy to remember, but the trouble is that when you go off abbreviating everything, and everything gets to be a lot of things, I soon forget just which letters I left out. I find it a whole lot simpler to type the whole command out:

list = ls
superuser = su
remoteshell = rsh
remove = rm
remotecopy = rcp
copy = cp

it's more typing, but it is so much easier to remember, to read and to guess when you don't know, than all of the abbreviated versions. And besides, that is why they came up with graphical UIs, so we would not have to type (or remember) anything at all!

I know that the Linux community disagrees with me on this (GNOME users notwithstanding), but I just had to vent somewhere.

{/soapbox}

Now I can get on with what I really like about Linux: scalable, cross platform, stable, multitasking, vaguely real time, small footprint, massive number of device drivers, etc...

I truly believe that the embedded landscape would be in a world of hurt if not for Linux. The processors have grown in power so much, and out demands of embedded systems have increased so much that we need a real OS. Simple state machines and hand rolled task swappers just don't cut it when you need to use a hard disk the store and retrieve data, or use a PCMCIA 802.11 card and TCP/IP in your embedded system. I'd like to see what a pain in the #$% it would have been to implement the empeg on a CE platform. I wonder if a Hijack trick would have even been possible, and would I have to keep installing security patches on it?

{soapbox}

Now would someone like to get me started about this "we can make stuff just as good as or better than Micro$oft in the free arena" dollar sign thing?

{/soapbox}

... damn soap box, I keep stepping on it ...

BTW, I could still use some help on Q2 and Q3 if there is anyone left who does not hate me now!


Steve

(note to self: never attack Linux on the empeg site if you want some help... or at least log on as a different user...)

So, my very first Linux script: "untargz" goes something like this:

!/bin/bash 

gzip -d $1.tar.gz 

tar -xvf $1.tar 

there is probably a better way, but this made simple (KISS) sense to me.

Well, yeah, there is a better way to do that: use "tar -xvzf $1" instead of the gzip and tar lines (and pass the complete filename, be it .tar.gz or .tgz). This way, you aren't using up that much disk space for the temporary file. Or you could emulate the way "tar -xvzf " by doing "gzip -c -d $1 | tar -xvf -".

Something that would be much simpler and make more sense to me would be something like:

copy /mnt1 somehost/mnt2 --preserveall

Well, if you replace "copy" with "rcp" (for Remote CoPy), you basically get to that function. "scp" (for Secure (shell) Copy) from the ssh package would be better though, at least when you are doing it through an untrusted network.

Somehow the rather cryptic 'tar' trick reminds me of the clever fellow who wrote an entire multitasking OS in one line of recursive C code (100,000 chars long and impossible to read or debug).

First: The tar trick isn't that cryptic once you got used to the basic unix tools and to the idea of pipes. Second: I am almost 100% sure that guy was (a) a nerd and (b) using some tool to obfuscate his source. Third: Obfuscated C is next to nothing compared with obfuscated APL. There has been an obfuscated APL contest (target: the most obfuscated APL program that returned "42" and didn't contain more than one single digit which had to be "0" if used at all), where the winner was unable to even recognize his own code after the judges removed the commentary lines at the beginning of his program.

Ok, this should be all I ever wanted to say regarding your Q1. I will now try to answer as much of Q2 and following as I can (but I don't use Mark's cross-compiler, but a previous version which had different executable names for the tools).

Basically, I can only answer Q3 because of that:
Try building the beep.c source that I attached to this post. It is simple and produces an output that you can surely recognize: sends a beep out through the normal audio path (i.e. through the EQ, RCA, amp, speakers).

cu,
sven

Note: I did not write the beep.c program, but downloaded it from somewhere (probably. from the old developer site).

Sven,

Thank you for your response. I was quite sure that after I "went off" a bit that I would be ostracized from the bbs.

Yeah, after I read about the 'z' parameter in the post by genixia and your subsequent "Well, why exactly you using the 'z' command line option there?", and of course after I "went off", I figured out that I don't really need a script and that TAR can handle the whole thing (unless its bz2).

Well, if you replace "copy" with "rcp" (for Remote CoPy), you basically get to that function. "scp" (for Secure (shell) Copy) from the ssh package would be better though, at least when you are doing it through an untrusted network.

You mean something like "rcp -pr /mnt1 /mnt2"? And you still contend that "(cd /mnt1 && tar cpzf - .) | (rsh somehost '(cd /mnt2 && tar -xvpzf -)')" isn't cryptic in comparison? I'm not even trying to say that pipes are not wonderful and powerful or that I can't follow what's going on there, its just that one of those looks simple and easy to follow, the other takes a bit of looking at.

I generally think of obfuscation as another term for someone's "job security" or just cheap vindictiveness. I picked up a contract completing an application where the programmer quit for a better job. The wanker left perfectly compilable code, but he stripped out every single comment, removed most of the linefeeds, all of the formatting and changed every variable into things like "dfg5yh65nb86sdf4". It took about 2 weeks just to make it readable again, before I could even give a good estimate of the time to complete the job. What a pain.

(but I don't use Mark's cross-compiler, but a previous version which had different executable names for the tools).

A previous version of Mark's cross-compiler?

Anyway, thanks for beep.c. I'll try it tonight. My plan seemed like a simple one:
1. Install the "precompiled toolchain"
2. try some simple thing to prove that it works.
3. try to compile a straight up kernel
4. figure out the .zImage thing
5. upon success to this point, attempt to obtain all of the pieces and build the toolchain myself
6. try some simple thing to prove it works
7. try to compile a kernel with my toolchain
8. try out some of my own kernel hacks
9. conquer the world. End violence, disease, hunger
10. figure out what to do about over-crowding

It's a good thing I'm out of work just now, as I plan to complete the list by the end of the week!

Thanks,

Steve

My plan seemed like a simple one:

1. Install the "precompiled toolchain"
2. try some simple thing to prove that it works.
3. try to compile a straight up kernel
4. figure out the .zImage thing
5. upon success to this point, attempt to obtain all of the pieces and build the toolchain myself
6. try some simple thing to prove it works
7. try to compile a kernel with my toolchain
8. try out some of my own kernel hacks

OK, so I launched off on this simple plan. It started out pretty good, but I'm having a little trouble along the way. Maybe someone can give me a few pointers...

1. Installed "precompiled toolchain" without any problems.

2. Built beep.c without problems. Noticed that I can build it with the path "user/local/armtools-empeg/arm-empeg-linux/bin" and the command "gcc" OR the path "user/local/armtools-empeg/bin" and the command "arm-empeg-linux-gcc".

3. and 4. downloaded the v2b11 kernel source from the empeg site (actually the beta 11 kernel source and the 14 patch files to bring it up to v2b11). After some twiddling with the unpack destination directories I got the kernel source unpacked and patched to v2b11.

I actually ended up with the whole thing in the following path:

"home/steve/download/empeg-kernel/empeg/kernel/..."

Q1: What path should I be using? The patch files really insist on the "./empeg/kernel" portion, but I gather from some readme files I read somewhere that the "home/..." area is not the correct place.

The Makefile uses the "arm-linux-empeg-gcc" stuff, so now I gather that the correct path for the cross-compiler bits should be the "user/local/armtools-empeg/bin".

I also see from the readme in that came with the kernel source download, that I am supposed to do the following (presumably from the ".../empeg/kernel" directory):

make configure
make dep
make zImage

Well, the "make configure" asks about a zillion questions. At first I thought that I could just take all of the defaults (since the makefile uses 'arm-linux-empeg-gcc', I assumed that the defaults were empeg). Not so. So I poked around and found lots of files with names similar to 'config' scattered throughout the kernel source.

Q2: Is there some kind of configuration file that answers all of these questions to give me a standard empeg kernel, or do I just have to go through each one and figure out the correct answer for the empeg? (many are obvious, but not all of them)

I tried to answer all of the questions by hand, but I must have messed up because the "make dep" had a few complaints but finished OK, but the "make zImage" failed miserably.

Can anybody point me in the right direction?


Thanks,

Steve

Anyone else feel free to stop me and tell me I'm doing it all wrong but this is how I do it. Don't bother patching things up to 2.00 b11. Just download it....

0. I am assuming you have extracted the toolchain correctly here and have a correct path. My path is "PATH=/usr/local/armtools-empeg/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/christianh/bin" and works fine for me. As you already have noticed, the compiles use the arm-empeg-linux-sometool versions.

1. Download the v2b11 kernel tree from Mark's Site (here)

2. Download the three patches:
rdsfake
voladj
hijack 265 (or later if there is one)

3. Unpack and untar the bz2 linux kernel archive -> anywhere! That's crap about it being in a specific place. It will create a linux-v2.00b11 directory in your current path (bunzip2 and tar xf <file> works fine for me)

4. Change the name of that directory to whatever you want like "mykernel" or something.

5. Change into that directory so you are now in /whatever/path/you/like/mykernel

6. Apply the patches. These work a little different because Mark does his diff differently (keh?) So...
patch -p0 </path/to/voladj.patch
patch -p0 </path/to/rdsfake.patch
patch -p1 </path/to/v265.hijack.v200b11.patch

The -p1 means remove the first level of the path information.

There will be no patch errors. If there are something is wrong.

7. You now have fully patched hijack kernel.

8. Copy the default config in...
For a Mk1 kernel
cp arch/arm/def-configs/empeg-car1 arch/arm/def-config/empeg

For a Mk2 kernel
cp arch/arm/def-configs/empeg-car2 arch/arm/def-config/empeg

Edit : That'll teach me to do this off the top of my head. Missed a step between 8 and 9

8.5 Run "make empeg_config"

9. Now run "make oldconfig". This will answer all but one of the zillion questions you got when you ran make configure. The one you have to answer is the command line. Simply press enter because the empeg kernel ignores it anyway.

10. Then run make dep; make clean; make zImage

11. Upload your shiny new kernel from arch/arm/boot/zImage

HTH. Let us know. This works fine for me but as the old saying goes your mileage may vary. We'll worry about doing diffs and patch files once you get this done.

Tony or perhaps Loren (Hijack FAQ guy?) could this go in the FAQ somewhere. It took me and I know others a lot of time figuring this stuff out. If I had a step by step instruction list like this...

Aaahhhh crap.

That'll teach me to do this off the top of my head. Missed a step between 8 and 9

8.5 Run "make empeg_config"

yes! That rocks!!! Successful kernel build in da house! Thank you Shonky (that's exactly what I needed) and SMU and of course Mark Lord!

So, for fun (and education) I wrote the following scripts:

One to download and install the precompiled toolchain: 'gettools'
#!/bin/bash
#
# This will download the precompiled toolchain form Mark Lord and install it in /usr/local/armtools-empeg
#

( ! test -f armtools-empeg.tar.bz2 && wget http://empeg-hijack.sourceforge.net/armtools-empeg.tar.bz2 )
su -c "(( test -d /usr/local/armtools-empeg && rm -r /usr/local/armtools-empeg ) ; tar -xvjf armtools-empeg.tar.bz2 -C / )"


One to get all of the sources: 'getsource'
#!/bin/bash
#
# This will download:
# the unmodified v2.00b11 source,
# the voladj.patch,
# the rdsfake.patch and
# the v###.hijack.v200b11.patch
#
# all from Mark Lord's site.
#

echo v${1:-265}.hijack.v200b11.patch
( ! test -f linux-v2.00b11.tar.bz2 && wget http://empeg-hijack.sourceforge.net/linux-v2.00b11.tar.bz2 )
( ! test -f voladj.patch && wget http://empeg-hijack.sourceforge.net/voladj.patch )
( ! test -f rdsfake.patch && wget http://empeg-hijack.sourceforge.net/rdsfake.patch )
( ! test -f v${1:-265}.hijack.v200b11.patch && wget http://empeg-hijack.sourceforge.net/v${1:-265}.hijack.v200b11.patch )


And one to unpack, patch and build the kernel (it preforms all of the steps outlined by Shonky): 'extract-patch-build'
#!/bin/bash
#
# This will extract the kernel source, apply the patches, and build the zImage
#
( test -d linux-v2.00b11 && rm -r linux-v2.00b11 )
tar -xvjf linux-v2.00b11.tar.bz2
cd linux-v2.00b11
patch -p0 < ../voladj.patch
patch -p0 < ../rdsfake.patch
patch -p1 < ../v${1:-265}.hijack.v200b11.patch
cp arch/arm/def-configs/empeg-car2 arch/arm/def-configs/empeg
PATH="/usr/local/armtools-empeg/bin:$PATH"
make empeg_config
make oldconfig
make dep
make clean
make zImage
echo
echo Your kernel image is in linux-v2.00b11/arch/arm/boot/zImage
echo


These can be run from any directory. For example, mine are in '/home/Steve/download/empeg/hijack'.

These download and build hijack version 265 by default, if there is a newer version then type the version number on the command line like this: './getsource 266', and './extract-patch-build 266'.

So if you were starting from scratch you could:

1. create a directory like ' /whatever/path/you/like/mykernel' and cd over to it
2. place the three scripts in that directory
3. run the './gettools' script and supply your root password when prompted
4. run the './getsource' script
5. run the './extract-patch-build' script pressing enter when it pauses

and there you have it! All of the tools, source, patches and a nice shiny new zImage all generated on your Linux machine.

I'm sure there are better ways of doing things, but this gets all of the pieces, puts them in a usable place, applies all of the patches, and performs all of the build steps. So it's a good start that answers many questions.

So, from here I can begin to hack the kernel, and if I screw it up too badly, I can always go back to ground zero fairly easily.

The next most obvious question is how do I create my own '.patch' files?


Thanks,

Steve

The next most obvious question is how do I create my own '.patch' files?

> man diff

;-)

You mean something like "rcp -pr /mnt1 /mnt2"? And you still contend that "(cd /mnt1 && tar cpzf - .) | (rsh somehost '(cd /mnt2 && tar -xvpzf -)')" isn't cryptic in comparison? I'm not even trying to say that pipes are not wonderful and powerful or that I can't follow what's going on there, its just that one of those looks simple and easy to follow, the other takes a bit of looking at.

The two options here are to have a collection of utilities that can be combined in useful ways or to have a separate application for every possible situation.

I would contend that having a small set of tools and some intelligence is much better than having to remember all 10,000 tools I might need.

Microsoft would seem to disagree.