Embedded PCI and Linux

In the 2.0.38 kernel we used as a starting point, Linux support for PCI peripherals is extensive but is completely dependent on BIOS calls for PCI configuration. This works in the world of PCs but is a gaping hole in implementing PCI for an embedded system. We were unaware of any Linux implementations of PCI that did not use the BIOS, and so our starting point was to give our system PCI BIOS functionality in order to make use of the base Linux PCI driver.

Most developers are familiar with PCI in the Intel x86 environment. It is important to note that such an implementation is a special case. In the special case, the PCI memory address space and the host processor (386, 486, Pentium, Athlon), physical address space are one and the same. The bridge arrangement required by the VZ328 architecture described above is the more general case.

There are other significant development issues as well. These stem from the reality that most PCI development is for products intended for the desktop. In addition to the above problem, other problems involved in porting PC drivers to our embedded platform include: 1) byte order is generally ignored as x86s and PCI are the same; 2) word alignment is generally poor on x86s as these processors automatically generate multiple bus cycles; 3) new hardware still reflects ISAs (industry-standard architectures, i.e., AT) by using the same I/O register maps, rather than introducing improved memory space register maps; and 4) drivers, especially video drivers, sometimes rely on expansion ROMs that contain x86 binary code, which assumes the existence of PC peripheral devices and specific x86 PC BIOS software interrupt vectors.

The consequence of our decision to use the Motorola VZ328 microprocessor is that placing the memory onto the PCI bus would transform the architecture into the special case of the PC, at increased cost and great technical complexity.

The PCI BIOS must do two things. It must allocate memory space, I/O space and interrupts. It also must allow a device driver access to PCI configuration cycles that allow the device driver to find the card, read the card resources and be able to configure the card as necessary.

The PCI BIOS calls have been extended to provide a hardware abstraction layer to compensate for having a split memory address space. These functions are summarized as follows:

The VZ328, unlike x86 processors, does not have both a memory address space and an I/O address space. The VZ328 itself cannot generate an I/O cycle, nor does it need to. The VZ328 must communicate the PCI I/O address to the bridge, as required for PCI memory addresses, and must communicate a request to the bridge for a PCI I/O cycle. The PCI BIOS calls have been extended with the addition of the following functions to support I/O cycles:

The Cypress Cy7C09449 contains a 32KB dual-ported memory that is used to bridge the VZ328 memory space to PCI memory space. Since there may be more than one PCI device in a system, there can be more than one PCI device driver installed at any given time. This condition compels us to treat the dual-ported memory as a shared resource and to balance efficiency with performance.

The kernel function kmalloc(), which calls get_free_pages(), was designed to allocate such special memory using GFP_ flags. GFP_DMA is defined for PC implementations and denotes a page that is bounded by a 16M memory limit and does not cross a 64K boundary (this was due to the limitations of the 8237 DMA controller at the time).

Since PCs share processor and PCI memory address spaces, there is no special flag defined to select memory for use with DMA and PCI. It appears that the PC standard here is designed around the special case. To compound the problem, there is no central resource to initiate a DMA transfer, as device drivers initiate transfers by writing to nonstandard registers on the device hardware. Generally speaking, this code needs to be altered to become bridge-aware. Therefore, to facilitate sharing the dual-ported RAM on the bridge, the PCI BIOS has been extended to include kmalloc memory tables and a GFP_PCI flag. A device driver may allocate and deallocate DPram on a transfer-by-transfer basis or request a block of memory that it will own permanently, until the device driver is closed.

Finally, interrupts are handled using the register_interrupt() system call. PCI specifies an 8-bit field for interrupt number, so the VZ328 PCI BIOS can assign a mach interrupt number that can be handled by a device driver using the register_interrupt() system call.