How to Port Linux When the Hardware Turns Soft

Porting Linux to run on the Pico E12 and beyond.
Getting a Pulse

My first objective was to write the proverbial “Hello World” program for the E12. I spent a few days and wrote two different “Hello World” programs: one for the keyhole FIFO and one for Xilinx uartlite port.

Now, I was ready to attack Linux. I decided to start with Linux 2.6. There were numerous issues—good reasons, as well as respected and conflicting opinions favoring both 2.4 and 2.6. I elected to use Linux 2.6, because I eventually was going to have to move to 2.6 anyway. Initially, I used the PowerBook to configure and build my Linux kernel for the Pico E12. This allowed me to start without cross compilers. Eventually, I switched to building inside of a coLinux virtual machine on Windows hosting the E12. Most Pico clients are doing Windows-hosted development. It was critical that everything work in that environment. Besides, building a PowerPC Linux kernel in a Linux virtual machine running Windows and loading it into a PowerPC, means that Linux outnumbers Windows 2 to 1 inside my laptop.

I used the Xilinx ML300 as a template to create a new Linux BSP (Broad Support Package). I grepped the kernel source for all references to the Xilinx ML300. I copied and renamed all ML300-related files to new files for the Pico E12. There were four completely unique files for the E12:

  • arch/ppc/platforms/4xx/pic0-e1x.c: board-specific setup code.

  • arch/ppc/platforms/4xx/pic0-e1x.h: headers and data structures for the board-specific setup code.

  • arch/ppc/platforms/4xx/xparameters/xparameters_pic0-e1x.h: a set of hardware definitions created by the Xilinx software that created the bit image for the FPGA.

  • arch/ppc/configs/defconfig_pic0-e1x: a default Linux configuration file for the E12.

There were major similarities between the Xilinx ML300, but there were a few specific differences. The E12 deliberately implements a lot less hardware. The E12's purpose is to provide a very minimal base platform, with the largest percentage of FPGA left for the client. The minimal useful Linux configuration must have either Ethernet, a serial port or the keyhole port. The default E12 does not have an interrupt controller—the PPC405 provides a timer interrupt that does not require a PIC. The E12 also uses the Xilinx uartlite uart, not the much larger and more common 16550 uart. There were no Linux drivers for the uartlite. Two other ML300 files, generic support for Virtex FPGAs required minor modifications.

The next major issue was learning the Linux configuration system. I was not able to find much documentation. With Linux kernel programming, the two primary resources are the Linux source itself and the mailing lists. Sometimes, there is excellent documentation for a system; sometimes there is nothing. Sometimes I found documentation in some obscure corner of the Web—after I had figured things out on my own. I had to develop enough understanding of the kernel build system to add a new board, some new configuration options and a few new drivers to the build system.

The first element is the Kconfig files in most of the Linux source directories. Kconfig is a cross between a very, very simple scripting language and a menu construction language. The entries in Kconfig files determine the menu structure and choices that you get when you execute any of the Linux menu configuration build options—make oldconfig, make menuconfig, make xconfig.

I had to create a new menu item under the ppc 4xx menu for the Pico E12, menu items in the drivers/serial/Kconfig file for the uartlite and keyhole serial ports, and a small collection of menu items for other options. The syntax for the Kconfig items I needed to create could be easily worked out by inspection and a small amount of trial and error. I copied blocks for similar objects, made name changes as needed, and without too much effort, it worked. Inside the .config file, source code and Makefiles, the configuration items defined in Kconfig files are prefixed with CONFIG_. After the Kconfig entries were created, entries needed to be added to the matching Makefiles. This mostly involved copying similar objects and making name changes, and except for a few very complex choices, was pretty easy.

So far, I had done very little actual coding. Most of what I had done was remove ML300-specific code from the new Pico E12 copy. I also copied the Xilinx PIC driver and created a stripped-out dummy PIC driver.

I was now able to build a Linux kernel for the Pico E12, without serial or Ethernet drivers. I still needed to write two serial device drivers: uartlite.c and keyhole.c. I deliberately chose to use the 8250 driver as a template—8250s and their numerous successors are ubiquitous, probably making up more serial devices than all others combined. I assumed that the 8250 driver would be, by far, the most stable and well-debugged serial driver. Also, many 8250-based systems are known to have problems with interrupts, so I knew that the Linux 8250 driver had to work without interrupts. This turned out to be a bad choice. The Linux 8250 driver is probably, by far, the most complex Linux serial driver.

Eventually, I remodeled my drivers based on the m32r_sio driver. I did not know much about the m32r_sio, but the driver was clean and simple, had all the features I needed and none that I did not. I also had to create a set of serial port headers for the keyhole and the uartlite, defining the uart registers and the bits within the registers. I also modeled these directly on the 8250, which was a much better decision. I have been writing uart code, including software uarts, for a long time. Writing the device-specific code for the keyhole and uartlite was simple. Fewer than a dozen lines of code were needed to send and receive a character. The uartlite and keyhole, like most Linux serial devices, do not have modem control and operate at a single speed. The few lines of code needed to send a character were also useful elsewhere for debugging. The keyhole is not a real serial device, but it can be made to look like one to Linux and then used as a console when the E12 is hosted. This was very important.

Connecting a rat's nest of cables to the host computer and to the tiny external connector on the E12 for Ethernet and the uartlite serial port created problems. The time testing every cable connection to assure that one had not come loose prior to trying a new kernel was greater than the time writing and testing code. I wore out or damaged several external connectors before I was done. When using the keyhole, all the connections between the E12 and the host are internal. It was also useful to send debugging through one device using the other as the console. The keyhole had one other attribute that came in extremely handy—I could write 16- or 32-bit values to one register as a single output instruction and see the data on the host side. This was critical when debugging PowerPC assembly code. Inserting code to display a value or trace execution needed to be done using few instructions, minimal side effects and assuming very little was working. Outputting values directly to the keyhole port became my equivalent to flashing an LED connected to an I/O port. It was equally simple and slightly more powerful.

To some extent, all software development is working in the dark, but embedded board bring-up is particularly so. Output is a flashlight letting you see a little bit of what is going on. The E12 has provisions for JTAG debugging, either through the emulated parallel port or the 15-pin connector. The Linux kernel provides kgdb and xmon support. These presume support on the host side and working hardware and drivers on the target. Linux also provides options for outputting progress and debugging prior to loading the console driver. These were limited primarily to 8250-compatible uarts. I added uartlite and keyhole ports to the early text debugging devices. Aside from persuading Linux to use it, this primarily involved supplying a few lines of code to output a character. I have the skills needed to use debugging tools from logic analyzers to gdb. Most of the time, I find that sophisticated tools provide massive amounts of additional information, obfuscating the problem rather than revealing it. But debugging is a religious art with competing sects, each with their own dogma.

Once I had working output routines for the uartlite and the keyhole, a stripped version of the ML300 code for the E12 and modified Kconfig and Makefiles for the E12, I was ready to build a kernel and try it. The normal Linux build process for the PowerPC leaves a kernel image in ELF format in arch/ppc/boot/images as zImage.elf. I copied this from the PowerBook I was using to build Linux kernels onto the host computer for the E12, and I used PicoUtil to replace the current Linux kernel image on the E12 Flash. I used the E12's monitor to execute the ELF file. The Linux boot process is similar across platforms and boot methods. In my instance, the zImage.elf file loaded at 0x40000000 and started with a small wrapper that did some early hardware setup, decompressed and relocated the actual Linux kernel and then jumped to the early Linux setup code. I copied the simple character output routines for the keyhole and uartlite into the files arch/ppc/boot/simple/keyhole_tty.c and uartlite_tty.c, and these provided debugging output during the wrapper execution.

My first big problem was that the memory map of the E12 had the Flash starting at physical address 0 and the RAM at a higher physical address. Advice I received on the Linux PPC embedded mailing list suggested I really, really did not want to try to port Linux to a board without RAM at 0, if it was humanly possible to persuade the board designers to change the memory map. There have been previous and subsequent efforts to modify Linux to support systems where RAM does not start at physical address 0. I believe that is less of an issue now. Still, I took the advice, and after a few hours of begging, Pico agreed to re-organize memoryped to 0. The soft hardware meant that they were able to provide me with a new bit image with RAM mapped to 0 within a few hours.

For a while, I also ran my own customized version of the monitor program, passing a board information structure Linux expected with a small amount of information on memory size, processor speed and the mac address to use for the NIC. Eventually, these modifications were incorporated into the standard Pico monitor.

The best documentation for the boot process as it applied to my system was in the MontaVista comments at the top of xilinx_ml300.c. These did not cover the decompression and relocation wrapper, but exposed the rest of the boot process.

______________________

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix