How to Port Linux When the Hardware Turns Soft
The next significant problem was in arch/ppc/kernel/head_4xx.S. Here, Linux does basic MMU and exception handling setup, then uses an rfi instruction to transition from “real” mode to “virtual” mode and continue with the kernel initialization. I was able to execute right up to that rfi. I was able to check all the obvious conditions for successfully executing the rfi. However, I never ended up at start_here—where the rfi should have continued. I spent days developing an understanding of the Linux Virtual Memory system—most of the documentation x86-specific. And, I became more knowledgeable about the PowerPC MMU, a fairly simple device compared to the x86 MMU. It is basically a 64-entry address translation table. Virtual memory OSes inevitably use more than 64 virtual-physical addresses mappings, region sizes and privileges. A reference to a virtual address not in the MMU, or one that violates the privilege bits set for that entry, causes an exception, and it is the OS's responsibility to sort it out using whatever algorithms, methods and data that suits it. The fault processing might take longer, as it is not handled in hardware, but it is more flexible, adaptable and less resource-intensive. There are no gigantic fixed mapping tables in dedicated regions of physical memory, as required on some other processors.
But, I still could not figure out why the rfi was not executing correctly. I added all kinds of additional entries to the MMU, assuming that I was actually successfully switching to virtual mode but unable to communicate, because my I/O ports were no longer accessible. I sprinkled the equivalent of “I am here” debugging markers throughout head_4xx.S and got my first clue. I was continuously looping through an exception handler. Every time I switched to virtual mode, I lost control of the PPC, regaining it again in real mode in the exception handler. I had the critical clues to figure things out, but I was still mystified.
Every problem can be solved if it can be divided into smaller pieces. Eventually, I realized that it was possible to transition from real to virtual mode in smaller increments rather than all at once as the rfi did. I was able to turn on address translation for data and turn it back off without ill effects. I was able to add 1-1 physical to virtual address mappings for my keyhole debug port to the MMU, turn it on to do some output and turn it off. With more effort, I was able to turn on instruction address translation execute code and turn it back off.
That is when it finally dawned on me that the problem had nothing to do with switching from real to virtual mode, but that something else being set by the rfi must be enabling an exception that was not occurring otherwise. So, I tested the bits in MSR_KERNEL—the PPC machine status register value Linux uses—one bit at a time, until I discovered that anytime I set MSR_CE, enabling machine check exceptions, I lost control. I redefined the macro that set MSR_KERNEL so that it did not set MSR_CE for the E12 and reported to Pico that I thought there was a hardware problem in the E12. Pico never found the problem, but six months later, updates to Xilinx's firmware building blocks corrected the problem.
After working around the machine-check problem, I suddenly found Linux booting all the way through to setting up the serial/console driver. I was stalled for a few days while I actually finished the serial drivers for the keyhole and uartlite. Linux needs a place to hold the root filesystem. There are many possibilities. Frequently, the norm for embedded systems is to place the root filesystem for an embedded development environment on an NFS share on another machine. This requires a working Ethernet driver. My confidence in my serial drivers was not high at that point. Further, the Pico minimalist mantra does not include networking as part of the base Linux, and many E12/Linux applications do not need it.
The root filesystem can be on a hard disk (none readily available in the E12) or in Flash. The E12 uses a very simple Pico File System, but one that is not suited for a root filesystem. Another alternative was to put the root filesystem on a RAM disk. Linux provided the ability to use and populate a RAM filesystem as an intermediate step in the boot process. One objective was to migrate as much of the Linux boot code out of the kernel to user space as possible. Linux systems going back many years boot through initrd, then execute a pivot_root to switch the root filesystem from the initrd RAM disk to the disk-based root filesystem. Using initrd requires the loader to copy the compressed Linux image and the separate image of the contents of the initial RAM disk into memory, and provides Linux with a pointer to the initial RAM disk data.
Linux 2.6 introduced a new variation—initramfs. One difference between initramfs and initrd was that with initramfs, the contents of the initial root RAM disk filesystem were compressed into the Linux image during build, so there was only one file—in my case an ELF file—to load. This meant that the Pico monitor would not need changes. This initramfs approach proved to be extremely clean, simple and easy to use. Getting it working was complex and time consuming, because initramfs is fairly new. The primary documentation is a collection of posts to LKML. To create an initramfs for the Pico E12, I determined that I needed to create a directory on my build system and populate it with the files for the root filesystem. I enabled the initramfs option using menuconfig and told menuconfig where to find the directory that represented my root filesystem. There are a few other ways to do this, but that was the simplest. Initially, I decompressed the initramfs from the Gentoo Linux install on my PowerBook. I eventually switched to a cross-compiled BusyBox when I erroneously thought I might be having problems with my boot image, because the binaries were built for the PowerBook, not a PPC405.