What is an embedded solution and why would you build your ``better mouse trap'' using x86 technology to run your application on a Linux platform?

Embedded computer solutions are many things, but they all seem to have one thing in common: they are not desktop computers. They tend to be small, low power and able to operate in hostile environments. Embedded devices operate in both fixed and mobile locations; often facing extremes of temperature, vibration or other environmentally challenging conditions.

In the not too distant past, embedded devices were designed around the least expensive microprocessor that would be reasonable, reliable and able to accomplish the task at hand. Commonly, these solutions where known as ``black boxes''. Black box devices did and still do provide effective solutions to meet the needs of various real-world problems. But, in essence, all embedded solutions have a number of things in common: the handling of repetitious tasks based on monitoring and/or collection of data, the storage of this data for future analysis, and simple to very complex control functions based on software algorithms which compare the data stream to pre-defined set-points.

Figure 1. Segmentation of Embedded Market

It is estimated that the world-wide market for embedded solutions will top 20 to 30 times the total dollars spent to date on the desktop computer market (see Figure 2). It is not a coincidence that this pending explosion of an estimated three billion dollar per year industry directly coincides with higher yields from the chip manufacturers, much higher gate densities on silicon wafers, and more and more functionality being placed into single chips. The result is an incredible reduction in the cost of CPU horsepower. Add inexpensive x86 processing horsepower to the now ubiquitous deployment of the world-wide data communication pipeline called the Internet, factor in the foreseeable convergence in the near future of affordable high-bandwidth wireless data networks, and you have all the makings for endless opportunities for those with vision, technical skills and entrepreneurial desire.

Figure 2. Total Computer Systems

High numbers of these embedded solutions will use an Intel x86-based core as the foundation hardware layer. Many of you may ask why use the x86 with all it's inherent limitations. The answer is simple: with the introduction of devices such as the MachZ 586 133MHz CPU/FPU, which integrates almost all the needed functionality of a complete computer system in a single chip package, yesterday's ``project stopper'', the cost delta between MPU and CPU, is no longer relevant. Add to this a fantastic reservoir of x86 development tools. Add an equally amazing Linux operating system which is stable, reliable, supportable and cost effective. Add again a very wide and deep river of skilled, knowledgeable people, and you have the recipe and ingredients to bring new solutions successfully and quickly to market (see Figure 3).

Figure 3. Embedded Software Market

This is the first of a series of three articles in Linux Journal which illustrate how the MachZ (see Figure 4) chip used on the MZ104 board may be used to design embedded Linux applications. In this first article, we discuss the hardware aspects, introducing you to the MachZ Chip, the Tri-M MZ104 PC/104 MachZ board(s), and the PC/104 family of solutions.

Figure 4. The MachZ Chip

In the second article we will look at issues surrounding the creation and debugging of a small Linux kernel using a standard desktop host computer. The kernel will be targeted to run on the MZ104 single board computer. The second article will detail how to install and debug (via the host) an embedded version of a bootable Blue Cat Linux kernel and associated applications. The kernel may load off the MZ104's IDE hard drive port, an M-Systems DiskOnChip solid state flash disk, or directly off the Flash ROM.

In the third article, we look at the benefits and illustrate how to use many features of the MachZ. These include ZF Logic and FailSafe, a patented concept included in every MachZ chip which allows the solution provider to recover BIOS, O/S and application software and reestablish full operation after an un-imagined catastrophic failure occurs.

In conjunction with these articles, Linux Journal is running a design contest based on the MachZ chip, the PC/104 MZ104 board, the M-systems DiskOnChip and the BlueCat Linux operating system. One hundred complete kits of these products will be awarded to the ideas judged best. Out of these winners, additional prizes will be given to those who can successfully develop a working proof of concept for each category entered. The free kits will include:

In June of this year, ZF Linux Devices Inc., released their single device PC-on-a-chip, the MachZ. This device heralds a new era in ultra-low power consumption utilizing standard x86 technology. Designed for embedded applications, the MachZ comes bundled with a Phoenix BIOS license and lists for $60.00 (US). In larger volumes, the price drops significantly to the mid-$30s.

The MachZ chip uses 0.25-micron technology and is packaged in a 388 pin Ball Grid Array (BGA) measuring 35mm x 35mm. Operating at 133MHz it consumes less than 0.5 Watts. The objective of this device is to shatter the price barrier, allowing developers to create very low cost Internet appliances and other embedded devices.

The MachZ chip combines full x86 PC functionality with its patented fail-safe technology into a single chip. MachZ is the only chip with a built-in FailSafe mechanism for restoring operation after a system failure. The MachZ chip runs all PC software without modification and it boots by simply applying power.

The MachZ is essentially an enhanced 586 CPU which will run at speeds of 33 to 133 MHz. It has a full 32-bit processor core including a math co-processor (FPU), and an 8kB L1 write back cache. It incorporates power management to further lower power consumption and allows the CPU be put to sleep and then awakened as needed.

In addition to the CPU, the MachZ incorporates many additional logic systems (see Figure 5). The North and South Bridge cores inside the MachZ generate full ISA bus and full PCI bus signals which allow standard devices to talk to the MachZ 586 CPU. The North Bridge controls and manages a fast (bursting) SDRAM interface. Not only is SDRAM the modern interface, but the MachZ has a significant cost advantage over the competition as it can use either 16-bit or 32-bit wide SDRAM chips.

Figure 5. MachZ Block Diagram

In addition to implementing the PCI bus, the South Bridge core also provides DMA and IRQ services, together with I/O controllers for the floppy, parallel, two serial, IrDA, AT Keyboard, PS2 mouse and I2C bus ports. As a bonus, memory chip selects, I/O chip selects, and general purpose I/O pins are brought out to further reduce the need for off chip glue logic.

The South Bridge core provides PCI Bus arbitration, allowing many devices to assume control of the bus. The chip supports up to three external bus masters.

The ZF Linux engineers designed many other useful features. The FailSafe Boot Up ROM (BUR) contains Z-Tag logic which uses a ZF proprietary 1.5MB 2-wire serial input. When the chip powers up, depending on the bootstrap jumpers, it will either start in the Flash or start in the on-chip BUR. The BUR will read and execute commands through the Z-Tag interface or the first serial port. A significant set of commands is understood by the BUR, allowing reprogramming of the Flash or FailSafe Recovery of a downed system.

The BUR has an interesting debug capability. Even with no external DRAM or SRAM connected to the MachZ, you can boot up the BUR and then download small test programs into the MachZ on chip RAM to verify the hardware components of your MachZ system. This debug capability and the FailSafe capability will be discussed in the second and third articles in the series.

But a chip, even one as sophisticated as the MachZ, needs some board real estate to sit on and bring out the many necessary signals and input lines. PC/104 has, over the last 10 years, been widely adopted by the embedded market due to its small size and rugged and reliable characteristics. An IEEE standard, PC/104 is an electrical and physical standard for the 8/16-bit ISA bus signals, using a pass-through header connector system on a printed circuit board of fixed dimensions. Also included in the specification are well defined locations for connectors. PC/104+ is a new addition to this original stand and, which maintains the physical size but incorporates, in addition, board to board routing of PCI bus signals.

There are hundreds of companies making PC/104 compliant boards. The companies range in size based on number of cards manufactured from one or two highly specialized offerings, to upwards of 100 different products. Unless you are looking for some truly esoteric plug-on card, you can pretty much buy off the shelf anything your specifications require. The MZ104 board (see Figure 6) manufactured by Tri-M is one such product competing for the attention and business of OEM application developers and embedded solution providers.

Figure 6. MZ 104 PC/104 MachZ Board

The connectors around the outside edge of the MZ104 allow for the hook-up of all the I/O cables. A cable kit comes with the contest boards. The hook-up includes up to 4 IDE devices (hard disk drive and/or CDs), a single USB, a keyboard, a PS/2 mouse, two serial ports, one parallel port, one floppy disk and power. The sockets shown on the bottom of the picture are the PC/104 8/16-bit ISA headers. These headers allow the boards to be connected together in a stack, with bus routing from card to card.

Where are all the components to deal with these devices and busses? They are inside the chip!

The MZ104 can be powered by hooking up a single +5VDC supply directly to the card or through the PC/104 bus connector from another board in the stack.

With all the different options available as a platform to build your better ``mouse trap'', why choose PC/104? Design criteria, which includes one or more of the following, are:

One of the very first companies to get working samples of the MachZ chip and to begin designing solutions based around this revolutionary device was Tri-M Engineering in British Columbia, Canada. Tri-M, an 18-year-old company, earns its living selling and supporting embedded x86 equipment to OEM and in-house solution providers all over the world. Tri-M would never have considered designing their own line of x86 single board computers until the ZF Linux MachZ came along. This gave Tri-M an opportunity to play at the table, which until the arrival of the MachZ was reserved for the big guys. Tri-M is now able to offer three different commercial off the shelf (COTS) CPU boards to its clients; in addition Tri-M has contracts to design two custom boards based around the same technology. Not a bad position to be in for a company basing it's future success on the success or failure of its client's products.

The MZ104 board is a fully PC/104 specification-compliant board, using the MachZ on a 6 layer circuit board. The MZ104 brings out most of the chip's internal features and supports a 2 or 4MB Flash ROM, the M-Systems DiskOnChip, and a 144 pin SO_DIMM connector supporting either 32 or 64MB of fast SDRAM. The MZ104 is the first and least complex member of a family of three PC/104 boards designed around the MachZ. This first board is the board which will be used in the design contest. Features of the board are shown in Figure 7.

Figure 7. Feature Chart of Tri-M MachZ-Based CPU Boards

The MachZ chip internally generates both the ISA and PCI bus. PCI is much faster than ISA, and the MachZ supports PCI bus mastering. The PC/104 standard does not allow interconnect of the PCI bus from one board to another, but devices on the CPU board may use the PCI bus. For example, the ICC104MZ uses on-board PCI for the 10/100Mb Ethernet chips.

Although there are few PC/104+ boards built, these ``new standard'' boards have an additional header allowing the PCI (as well as the ISA) bus to be routed between boards.

The MZ104 has a large white U-Shaped Connector which holds the SO_DIMM memory module (see Figure 8). The majority of the remaining circuitry on the board is for clock control. This very high integration of the MachZ chip provides higher reliability and greater MTBF. Failure rates for all electronics can be directly related to the number of discrete components. Less means more in terms of infant mortality and expectation for long life.

Figure 8. Annotated MZ104 Board

The MZ104 has not brought out all of the MachZ chip PCI bus signals. This was intentional so as to produce a minimum-cost board, free up board real estate for other components and to match market demands for embedded solutions which, for the vast majority of applications, don't usually require the high bandwidth of the PCI bus. To further save cost, all the components are placed on the top side of the board, reducing manufacturing cost and time.

A typical PC/104 stack of cards would include some of the components listed in Figure 9. Prices are given for small quantities.

Figure 9. US Dollar Cost of Typical Components

Figure 10 shows a MZ104 plugged into a three slot, standard 16-bit ISA bus passive backplane. Backplanes of this type are available from various manufacturers.

Figure 10. The MZ104 Plugged into a 16-Bit ISA Bus Passive Backplane

Backplanes such as this allow you to combine ISA cards and PC/104 cards during prototyping and testing. Although only the MZ104 card is shown in the figure, a complete PC/104 stack could be connected to up to three ISA cards using this backplane.

The backplane allows for easy testing of compatibility of features and functions in chip sets under consideration for design into your embedded solution.

When would you use a passive backplane? If, for example, you had five different frame grabber chip sets, you could purchase ISA cards which use these chips and then test and implement your software, without having to build custom boards of your own, using each of the chips under consideration.

The Tri-M DEV 104 Development Backplane Board (see Figure 11) allows not only the connection of PC/104 and ISA cards, but provides hardware prototyping areas for both thru-hole and surface mount electronic components. This is very useful for solution providers who require the development of custom boards based on a hardware prototype. In addition to the PC/104 header that the MZ104 is mounted on, there are three additional PC/104 connectors, one of which has male pins facing upwards which allows testing of components on the back side of a PC/104 card. That's much easier than testing in a stack!

Figure 11. The Tri-M DEV 104 Development Backplane Board

The PC/104+ specification is a superset of the PC/104 specification. The + specification adds an extra connector for the PCI bus. Both types of cards can be mixed in the same stack. This is done by putting the 104+ cards in the stack such that those cards having both buses are plugged together. Standard PC/104 cards may then be attached to either the top and/or bottom of the card stack.

The maximum distance between cards (due to pin specifications) is .6 inches. Older PC/104 CPU cards usually had to occupy the top position in a vertical stack to allow for heat dissipation. However, the latest generation of CPU cards can be placed anywhere in the stack due to their very low power consumption.

PC104 modules will stack together, connecting the bus from one card to the next. In this case, the top card is a relay board showing the somewhat industrial history of the PC/104 modules (see Figure 12). In the middle of the stack we have a CPU card, and on the bottom there is a UPS battery board.

Figure 12. Stacking PC/104 Boards

The PC/104 connectors themselves use long pins and deep sockets to provide a very reliable electrical interconnect and which act as a spine to strengthen the stack. Nylon or metal spacers are used on the corners adding strength and rigidity to the stack. Compare this to standard ISA cards, which are prone to intermittent electrical connections--especially when in motion!

Figure 13 shows an aluminum enclosure with rubber shock absorbing mounts designed specifically for hostile environments.

Figure 13. PC/104 Module Stack

Enclosures for PC/104 are basically there to protect the electronics from the environment. They are also designed to shed as much heat as possible to keep the internal electronics cool. Good designs also include consideration of vibration and G-force minimization to protect the PC/104 stack when mounted in high vibration or G-force locations.

Marketing people like the idea of selling a block box solution whose price is justified by its function rather than its component costs. These high stamina enclosures provide added protection and, thus, added value.

At some point prior to committing serious manufacturing money into your ``mouse trap'', or even a beta run of production models, a key question arises: do you build you own hardware from scratch or do you buy commercially available, off-the-shelf components or maybe some combination of the two? Assuming your proof of concept unit works and the guys with the money are hot to finance your dream, it's time to bring together the engineering and marketing staff and, as best you can, determine how many units you think you can sell during the window of opportunity.

From my experience in using x86 as the solution platform for the last 18 years, the economies of scale to convert from a PC/104 stack to a custom board was usually somewhere above the 5000 unit mark. I no longer think this is true today. With the advent of chips like the MachZ and others that are sure to follow, we have seen a truly new paradigm shift which will greatly benefit the smaller and wannabe electronic manufacturers. Using this new chip, the switchover point to develop a unique or custom hardware solution and to bring that device to a successful fruition has dropped to perhaps less then 1000 units. There are many reasons for this: we can get more functionality from this single chip, it uses less board space, it requires less engineering hardware design time so that designs can be accomplished in much shorter periods.

Further, the MachZ comes bundled with a fully licensed and integrated Phoenix BIOS. This alone significantly reduces the cost to roll and support your own hardware from scratch, and at the same time, effectively outsources any x86 software compatibilities to the chip manufacturer. This greatly reduces risk of software failures and the subsequent time delays to find and fix these bugs. This contributes to getting your mouse trap finished and into production while the market window of opportunity is still open. In the past, getting your hands on a major BIOS involved lots of costly legal issues, let alone negotiating a reasonable cost for less then 5000 units (a small quantity from the perspective of the BIOS manufacturers). Thus, the BIOS was often a show stopper for the small or low volume project being accomplished from scratch. Add to this the product life burden of maintaining both backward and forward software after you have spent considerable technical resources to adapt the BIOS to work properly in custom hardware configuration.

So where are some of the market opportunities, ripe to be harvested, using an embedded solution with a MZ104 hardware layer, a flash drive and the extremely reliable Linux operating system? They are literally everywhere.

Consider the area of Supervisory Control and Data Acquisition, SCADA. These systems are used extensively by power, water, gas and other utility companies to monitor and manage their distribution facilities. They also are used to monitor and control usage levels, load sharing/shedding and more and more for the purpose of remote meter reading. Historically, the hardware used in SCADA applications absolutely required the very highest reliability hardware available at the time. If was not uncommon to see a very expensive Micro VAX or other mini computers sitting in a remote station. Not only was the capital cost for this hardware a major consideration for the operator, OS and application software, custom application modifications and ongoing maintenance often exceeded the capital cost many times over.

It is now possible to achieve this same or better level of reliability using the MZ104, Linux and much less expensive application solutions. With the ever increasing pressure on these utility companies' bottom lines, embedded replacement solutions which can economically justify their cost in a few months make very attractive opportunities.

Recently, Tri-M Systems/Engineering provided one such OEM, a publicly traded manufacture of an ``UltraViolet Light'' (UV) sewage treatment system contract for a PC/104 based control system. The system was made from a combination of off-the-shelf PC/104 hardware together with a custom I/O board. The objective of this system was to gather data from various sensors monitoring the cleaning process and to automate and control the amount of energy used to produce enough UV to kill the bacteria. Of course, the correct amount of energy and UV needed changes quickly with rate of flow and density of particles suspended in the sewage. In the clients<\#161> older systems, the lamps where always run at their brightest levels as the only way of assuring whatever bugs present died.

Moving to an embedded SCADA solution allowed the clients using the system to gain full recovery of the upgrade cost in a few months, with additional operation energy savings over the next twenty years estimated in the millions of dollars. Adding smarts to this type of system worked out to be a big win for the company, one-upping all their competition. It was also a win for the clients who will require less public money to look after a problem. And it was a win for the environment as a greener solution to a serious problem.

The majority of the hardware layer was made up of off-the-shelf PC/104 hardware, consisting of a 486 CPU, a solid state digital disk drive card, 10Mb Ethernet card, and a PC/104 DC to DC power supply card (to ensure clean power to the stack). These cards were mounted on a custom designed carrier board which also contained various relay, communication devices and sensors. The application was written as JAVA applets running on top of a web browser. This effectively allowed the OEM to remotely upgrade and monitor their systems over the Internet, and at the same time gave the treatment plant staff a GUI-based interface to adjust set points and alarm conditions to suit their particular desires. The total time from start of the project to hand over of working prototypes was four months, including time for the custom board design and application software.

Other areas where embedded x86 solutions could run Linux are process controllers, medical equipment, money collection and ticket dispensing, GPS autopilots, and any place where the collection, storage and manipulation of data can stand an improvement in efficiency. With the low cost of today's components, ideas for high reliability products which justify a ``smart'' solution don't require a whole lot of imagination. Think telecommunication, robotics, Internet appliances, data loggers or anywhere data can be turned into information. Information is power, and power is closely associated with money.

Standards and Sources

Hardware Terms and Definitions

Engineering, successful distributors and manufacturers of x86 embedded hardware solutions for the past 18 years. Doug is also a founding director of the American Anti-Child Pornography Organization, helping to protect children from digital criminals (http://www.antichildporn.org/).