How to Port Linux When the Hardware Turns Soft

Porting Linux to run on the Pico E12 and beyond.
First Words

After this, I hit my next problem. Linux was booting all the way though to executing /init where it just stopped. I wrote a trivial version of ls and included it in the kernel, calling it prior to exec'ing /init. Everything was fine. But on exec'ing /init, Linux became deaf and dumb. Debugging can be particularly difficult when the horses look like zebras. I spent a lot of time tracing through the Linux exec process, which was remarkably ingenious in many instances, doing minimal work and loading a process through page faults. Unfortunately, this made tracing what was happening very difficult and led me once again to the (almost) erroneous conclusion that I had a virtual memory problem. I wrote a Linux version of “Hello World” in PPC assembler with no external libraries and was able to execute it as /init. But, I could not exec anything more complex. I eventually found and enabled system call tracing and was able to watch as /init executed. The system always died while in the middle of virtual memory operations. I ended up with failure cases when Linux would go dumb right in the middle of outputting some debug string—again, always during a VM operation. I could actually change the point of failure by inserting additional debugging. I was a victim of the Heisenberg uncertainty principle—observation changed the observed behavior.

I was sure something was wrong with my serial drivers, despite the fact that this did not make sense, but how else could output stop in the middle of a string? All the critical clues were present to solve this problem, though one of them was buried as an artifact of the machine-check problem. This was a VM problem, in a twisted sense, and it was a serial driver problem. I will not confess to how long it took for the answer to dawn on me. Let's just say I rewrote the serial drivers several times before I saw that although the serial drivers requested and saved a virtual address for the memory mapped hardware—partly as an error induced by using the 8250 serial driver as a starting point—the virtual address for the serial port was subsequently getting overwritten by the physical address of the port. Because in my efforts to debug the machine-check problem I put a 1-1 physical-virtual mapping directly into the MMU Translation Lookaside Buffer, I/O continued to work until the Linux VM system overwrote my temporary TLB entry. After recognizing this, it took less than 30 minutes to correct, and I was able to boot up Linux to a bash prompt.

The End—the Beginning

Little matches the thrill of seeing a new machine reach a shell prompt and knowing I made it happen. I had completed my base Pico E12 Linux port. Well, that is not quite true—no port is ever finished. When I completed my Pico E12 port, I was unaware of any other port of Linux to a Xilinx V4 chipset. Subsequently, a Linux port for the Xilinx ML403 by Grant Likely started working its way through Linux embedded PPC development trees and has been accepted into the distribution kernel. The Pico E12 is distinct from the ML403, but they are more similar to each other than the older ML300 from which I started. Grant's ML403 port reflected changes that were impacting the whole Linux PPC development tree, so I made my Pico E12 port track those developments.

Figure 3. Xilinx ML403 Development System

I have depended on the keyhole port for hosted development work, and as a result, the keyhole serial driver gradually has grown smaller and more consistent with the direction in which Linux serial drivers seem to be headed. I will have to update the uartlite driver to catch up.

I am currently on my third iteration of a Linux network driver for the E12, and Pico is on its second iteration of the underlying network hardware. The new network hardware is interrupt-driven requiring a PIC.

A second Pico board has matured, and with minimal changes, the Pico E12 port has evolved into the Pico E1X port.

Figure 4. Pico E14, Bigger, Faster, More Memory, More FPGA

I am working to get the Linux Memory Technology Device (MTD) system to work with the Pico Flash. This is complicated by the fact that the Flash in Pico hardware can be read and written to by both Linux and the host, and Pico is eventually planning on Flash device sizes that should be windowed into Linux memory instead of mapped in their entirety.

Once the Linux MTD work is completed, Pico wants a Linux (and Windows) filesystem driver for its simple filesystem—PicoFS.

Pico is considering changing the keyhole port so that on the host side (and possibly the target side), it sufficiently and closely resembles an 8250-compatible UART to use only the OS's native serial drivers.

Later, Pico developed a daughterboard for the E12 called the Little Brother Board that allows using the E12 in a non-hosted environment and includes three USB ports, an LED and several other hardware components. In one application, the E12/LBB combination is being used as a very high-performance Webcamera.

Figure 5. E12 in Little Brother Board Daughter Card

The E12 also can be hosted as a grid on a bus board called the supercluster. Currently, that configuration is used for blazingly fast code cracking, using FPGA hardware without OS support, but Linux HPC support is on the wish list. Higher performance can no longer be achieved simply by doubling the clock every 18 months. Clusters are a significant alternative; 16 E12s provide enormous horsepower while occupying little space and consuming little power.

There have been several iterative releases to Linux 2.6 since the E12 port, and occasionally, these require changes to the port.

I would like to get the Pico E1X port and drivers I wrote for it included into the Linux distribution kernel. Within the Linux embedded PPC mailing list, there has been some interest in seeing that happen. The code has been moved into git to make it easier to merge with new Linux iterations and to produce patch files for submission to LKML.