A few months ago, I began working at the National Center for Physical Acoustics on the University of Mississippi campus. The project I inherited was an embedded high-resolution SONAR that was applied in the fields of aquaculture and sediment transfer. For aquaculture research, the system is used to determine the biomass within a catfish pond. The system is able to resolve targets a few cm³ in air volume to centimeter range resolution out to a maximum range of about 100m (on a cool day). Our ultimate goal is to be able to target and size individual fish, though this may entail the use of multiple SONAR units coordinating several scans through a target area. Virtually identical hardware is used for sediment transport research. For this work, a pulse is transmitted, and the returned acoustic backscatter is analyzed to approximate the concentration of particulate matter per unit distance from the transmitter. Here, ranges are much shorter, usually no more that five meters. Both applications demand as focused an acoustic pulse as possible, so the system is tuned to operate at a relatively high 460KHz frequency.

The brains of the SONAR hardware, which I disassembled (see Figure 1), is a PC-104 stack consisting of components listed in Table 1. Both the motor and stack are encased within a waterproof (hopefully) canister. At the bottom of the canister is the acoustic transducer that serves to both transmit the acoustic ping and receive the return echo. The transducer must be mechanically rotated with a stepper motor. An umbilical cord carries power and network cables to the system from the shore. The original system used DOS on a CPU board that ran an executable which communicated with a host computer on shore. Users on the host ran a Java-based program on the host that controlled the SONAR using a 10MB Ethernet connection.

Figure 1. The Disassembled Aquascanner SONAR Showing the PC-104 Hardware Stack and Transducer

Table 1. Hardware Used in SONAR Linux Development

The system had passed its proof-of-concept design for both the aquaculture and sediment transfer experiments. As I began my work, modifications were being made to the hardware. In addition to improving custom SONAR receiver and transmit boards, a new in-house data acquisition board would be added. The old acquisition board, an RTD DM6420, would still be retained for use in an experimental system. I knew of DOS's weaknesses, particularly in the area of networking capability. I was interested in trying Linux in this embedded system, if for no other reason than to avoid DOS's kludged networking. It happened that waiting for our custom boards to be readied gave me enough time to experiment with a Linux-based software system.

I had been using Linux for almost three years on my desktop, but had no experience with Linux in an embedded system. I found through a glance of web resources that few hardware manufacturers officially supported it. I was worried that, somewhere down the road, a serious problem would arise between Linux and some sort of hardware. I determined that the safest way to proceed was to use hardware and software that would be compatible with both DOS and Linux and to develop both systems simultaneously.

My first step was to see if I could gather software tools that would work with both DOS and Linux. This way I could easily transport code between the two systems. Naturally the choice was gcc for Linux and the DJGPP compiler for DOS. Since my program would make heavy use of network programming, I needed both systems to be network capable. Linux comes that way, but not so with DOS. DOS needed both a hardware dependent network packet driver and a BSD socket compatible network library. After some work, and swapping of cards on the embedded stack, I managed to find an Ethernet card that DOS liked, and I installed the Watt32 DOS networking library. Familiar UNIX network programs such as ping now worked on either platform.

The first PC-104 stack I used had five boards: a power regulator, a network card, a VGA adapter, an IDE adaptor (this CPU board did not have an IDE header) and a CPU board with a 2MB M-Systems DiskOnChip. I installed DOS 6.22 first. I have to give credit where credit is due--DOS is easy to install and fits easily into the 2MB flash disk. Now, though, I was entering uncharted waters.

I expected getting an embedded Linux up would be a bit more difficult than installing a packaged DOS. I could have downloaded several versions of embedded Linux, but I had concerns about these releases. Many of these embedded systems use older kernels, and I knew I would have to write at least one device driver although I have experience with only the 2.0 and above kernels. I also wanted to use the new kernel so I could eventually take advantage of features such as PCMCIA and Firewire (maybe not for this project, but who knows?). In addition, I considered this a learning experience, so why not just try and come up with my own embedded Linux?

I used several computers for development including a 800MHz PIII and a 200MHz PII. The PIII motherboard has no ISA slots, so I cannot do any PC-104 hardware development on it (PC-104 is just a miniature ISA bus) but I can use it to compile my kernels and build a file system. The PII was used for hardware development. I have some PC-104 VGA cards so that I can plug a monitor into my DOS embedded system (remember, no serial terminal in DOS) and test out my embedded system.

My embedded development was done first with SuSE 6.3, then later with SuSE 6.4. On the Pentium III, I set up an additional 200MB hard drive mounted on /embedded and installed a minimal SuSE 6.3 with kernel sources using Yast's ``Install to Directory'' option. You could use any version of Linux you like, but I found SuSE's flexibility in installation a big plus for this work. I also added some 100MB partitions mounted as /emb1, /emb2, etc., to hold my embedded file systems. The plan was to mount these embedded partitions and copy the necessary components from my desktop to the embedded system. I also expected that I would need to patch or modify my embedded kernel sources at some point, and I used the source tree under /embedded for this purpose. I could still configure and compile the kernel as normal (make config, make menuconfig, etc.), so long as I had run chroot /embedded beforehand. Now I could build my embedded kernels and use LILO to boot the different kernels and file systems. I was ready to begin!

The embedded Linux Web site (see Resources) has several good sources for building Linux systems from the ground up. The ``Linux From Scratch HOWTO'' and ``Bootdisk HOWTO'' were both very helpful in building a bootable file system. M-Systems also puts out a manual ``Using the DiskOnChip with the Linux OS'' which is a good reference on populating a bootable root file system. M-Systems also provides a script with their DiskOnChip binary driver, copy2doc, that copies the basic files needed for booting your system. Keep in mind that this script is customized for Red Hat 5. With any other distribution, you will need to modify the script. Also, text listings of file systems are often found on the Web. Using the afio utility, you can also build a file system archive from a text file by running afio -ov embedded.afio < file_list.txt. You can then extract the archive to the working directory using afio -iv embedded.afio. Again, the same warning applies. You will need to edit the file listing if your libraries or directory structure are different from that of your file list source. Since my file system was pretty simple, I just copied everything manually (learning experience, remember).

The trickiest part of the entire process is getting the libraries right. The method I used was to run ldd /bin/* > libs.txt for each binary directory and check off each library as I added it. Brute force, but effective. The most troublesome libraries were the components necessary for PAM authentication, which I needed since this is a multi-user system. Here I ran into problems with the ``Go away, you do not exist'' error. I eventually found the culprit, but only after some trial and error.

The last step in building an embedded file system is to strip the debug code out of the libraries and executables using objcopy. This gives you a big space savings for your file system. For example, running objcopy -g libc.so.6 reduces libc from 4MB to just over 1MB. Definitely worth the effort! In the end, my file system was about 12MB in size. It includes full networking capability including telnet, ftp, and SAMBA; some hardware device drivers; the ash shell (much smaller than bash); and my application-specific binary code.

Now I needed to compile an embedded kernel. I did not include any functionality for a solid state disk as I was still booting from a hard drive, but I did use menuconfig to strip as much out of the kernel as possible. Without too much effort, I got a 2.2.13 kernel to 475KB in size, and I am sure I can find some more kernel features that I can cut out. After testing the kernel and file system on the desktop, the hard drive was put into the embedded system. A flick of the switch and a login pops up. Success!

Now I had to put the full Linux system and kernel on the embedded flash disk and configure LILO for booting. Never one for close tolerances, I chose a 64MB M-Systems MD2202 DiskOnChip. M-Systems' web site has binary drivers available for either 2.0.x or 2.2.x kernel versions. Following M-Systems' Linux installation manual, I edited some source files and compiled the kernel without major problems. One concern I had was with these binary drivers, but the open-source Memory Technology Device group offers an alternative driver (see the sidebar) which I will use in future development.

I planned on using M-Systems' ISA evaluation board to install the DoC first into my Pentium II. Booting Linux off the hard drive, the DoC should appear to fdisk as a block device in /dev/fla. I would use fdisk to partition the DoC, then build and copy the file system. I installed the DoC ISA board and booted Linux only to have a problem--I couldn't get Linux to recognize the flash disk upon boot. After several more kernel compiles and some e-mail to M-Systems, I found that the MD2202 is NOT supported by the binary driver. I traded my DoC for a 72MB MD2201, which M-Systems assured me would work. Upon installation, Linux immediately found a /dev/fla at boot. Running fdisk and mkfs on /dev/fla1 produced an ext2 partition that I mounted to /DiskOnChip. I could read and write to this device like any other drive.

I need to point out some differences between a flash disk and a regular hard drive. First, a flash disk has a finite life in terms of the number of times it can be written to. From an OS perspective, this means you cannot use swap space. It also means that you had better be careful with any of the write and fprintf statements in your code. What I have done is make the 20MB root partition on the DoC read-only. The rest of the disk is used for storing user binaries, and will not change too often. My 32MB of RAM is split into a 16MB RAM disk for data storage and file transfer, leaving Linux and any running software a comfortable 16MB of RAM. The RAM disk also helps to solve the second problem with flash disks: they are slow. The DoC2000 has a listed read speed of 1.4Mb/sec and a write speed of 500KB/sec. This means that using a 100MB/sec card for data transfer will do you no good if you are still reading and writing directly to the flash!

I loaded the DoC with my embedded file system by archiving my embedded file system using afio, copying it to the development computer with the evaluation board, and then extracting the afio archive to the DoC mounted on /DiskOnChip. I planned on booting the DoC with LILO, but LILO and the M-Systems' firmware driving the DoC try to share the same RAM space. To fix this problem, M-Systems includes a LILO update called plilo in the driver distribution. Again, following the M-Systems' manual instructions, I successfully made the necessary updates to LILO and the /DiskOnChip/boot directory. After powering down and removing the hard drive, I rebooted to find a LILO prompt and a functional Linux embedded system!

Now I began the build a fully functional stack for the embedded system. First, I needed to put the embedded system on the network. I plugged in a PC-104 10Mb NE2000-compatible network card and after a few tries configuring the autoexec.bat and config.sys files, I got the DOS packet driver loaded. Using the Watt32 utilities, I could ping computers from the embedded stack and run the old SONAR software. I had compiled ne2k support into the Linux kernel, so Linux recognized the network card without a hitch. Soon I could ping other computers...and run an ftp server, and run telnet, and SAMBA, all off an embedded stack.

The system also included two acquisition systems: a custom AD for field work and a commercial RTD DM6420 acquisition board for development work and lab experiments. To my dismay, RTD provides DOS drivers for the DM6420, but nothing was available for Linux. Needless to say, the custom board needed drivers written for both OSes. To write these drivers, I installed the PC-104 board into the Pentium II using a RTD DM16 PC-104 carrier board. This board allows any PC-104 card to be installed in an ISA slot. This way, I could work on my desktop system developing the drivers.

Our in-house board is a high-speed (for an PC-104 bus) acquisition card that needs a memory-resident driver to work properly; all of the outp, inp bit-banging user-space code that many people use for DOS device control won't work here. Here is why I dislike DOS. Even for a relatively simple driver such as the one for this AD, debugging is tedious (for starters, you must use software interrupts to even output to the screen), coding in a high-level language such as C is more trouble than it is worth (I just used MASM assembly), and there are not a great deal of resources or examples available. By comparison, writing device drivers for Linux is relatively easy--there are plenty of examples, good books (I learned with Allessandro Rubini's Linux Device Drivers), and the programming is logical and much easier to debug. I had finished the Linux driver for the custom board before I even got the initialization code to work for the DOS driver, and in the time it took me to finish and test the DOS driver, I had completed the Linux driver for the DM6420!

Now you may ask, "Okay, how did the thing work?'' I would have liked to give you an answer, but as of this writing, some delays in the building of a board have slowed the deployment of the new stack. In the next few weeks, I hope to have the system up and running in a test tank.

I can say that it did not take long to appreciate Linux's advantage in networking over DOS. Despite the BSD-compatible watt32, I had trouble with DOS's networking. Network code that worked fine when compiled under the gcc would not function properly when compiled under DOS's DJGPP. There is no substitute for having networking capability inherent in an OSes design. Linux's ability to run a telnet server will also prove useful, as I will be able to test software by running it locally on the SONAR--handy when there is little choice but to control the system remotely (to get an accurate idea of the transducer's response, the entire system must be underwater).

I believe that the DOS system will soon be scrapped. Future versions of the SONAR are going to have more custom boards, and I do not want to again face the problems associated with writing the DOS drivers for them. There will also be more network capability needed for building the distributed SONAR system, and I am confident Linux can handle the load. Linux's secure multi-user design also allows for easier administration of these remote systems. It's more than inconvenient when someone accidentally deletes a config.sys file on a DOS computer that is 100m from shore and underwater!

One area I did not explore in this work was that of a real-time Linux system. In its present state, the SONAR does not face timing problems crucial enough to warrant a real-time OS; however, future versions may. A distributed system is going to have to handle network commands, large file transfers, signal conditioning and data acquisition in a coordinated fashion. A real-time system may become a necessity. With Linux, I have the option of running a real-time kernel.

There are areas in which Linux faces disadvantages in the embedded market: I do not think Linux can ever have as small a footprint as DOS and still be usable. The features that make Linux so much more robust than DOS take code. Still, solid-state disk technology is improving both in terms of cost and performance. The smallest DiskOnChip sold right now is 16MB at about $4/MB. I believe time will soon make this disadvantage minor.

Another problem in the embedded Linux arena is lack of hardware support--the same problem that until recently had plagued Linux on the desktops. Here, the problem may not be as acute as with a desktop, as much embedded hardware is customized and the embedded designer must write the driver himself. However, custom hardware is expensive and time-consuming to develop, and designers will try to buy commercial products whenever possible. Linux needs more embedded suppliers to offer drivers for their products before it can become dominant in the embedded market.

The last problem with Linux is inherent in the design of Linux itself. Linux, unlike DOS, does not allow direct access to hardware. While a blessing to a networked server or desktop, this introduces some inconveniences to the embedded designer. Sometimes all an embedded controller really needs to do is strobe a port or handle an interrupt. Writing an entire device driver for such a simple task that could be executed in a few lines within a DOS executable seems like overkill. In Linux's favor, I would argue that for any production board that only needs some port oriented I/O, a 16- or even 8-bit embedded microcontroller board would be more far more cost effective than the simplest DOS computer.

My experience has shown that Linux makes an excellent embedded OS. Linux's networking capabilities proved better than those of DOS, even in the relatively simple networking application in this project. As more SONAR units are added to the system, networking will only become more complex, and I have confidence that Linux can handle the burden. I also like the flexibility inherent in Linux's design, allowing me to sculpt the kernel and file system into any shape I want. Not to be overlooked is the wealth of information available from other developers in newsgroups and on the Web for when problems arise. Whether a commercial version or your own, give embedded Linux a try!

Linux and the DiskOnChip

Resources