Reducing OS Boot Times for In-Car Computer Applications, Part III
This is the final article in a three-part series (see Resources) on reducing boot times for in-car computers. The impetus for the project has been my company, CarBot, where we're trying to get computer hardware to behave as it should in a car. In the previous two articles, I discussed a range of solutions for solving the problem of slow boots. In the months since those articles were written, we've thoroughly explored the possibility of using a BIOS replacement on the VIA Epia-M boards in an attempt to reduce boot times to the theoretical minimum. During this time, we've also discovered some hacks that can be used to minimize the effect of boot times on the user experience.
In our earlier articles, we compared the unnecessarily slow boot process to that of a car radio. The car radio boot times have climbed from nearly instant in the 1980s to several seconds today, but they still are rapid enough to be hardly noticeable. If you watch a modern radio right after you start the engine, you may see it do a small power-on self test, flash all the lights on the unit--much as dashboards in many cars do--and then power on the amplifier, producing sound within a second or two.
We decided to use the radio as our benchmark of rapid usability and appliance-like behavior. We attempted to minimize the following two variables: time from computer power-on to video and time from computer power-on to audio.
Our testing rationale was if we can minimize the time before the arrival of audio and video in the boot process, we later can optimize our software so that whatever the user was seeing and hearing last can be restored. This is what a radio does, this is what a television does, this is what a computer should do.
It takes users a couple of seconds to orient themselves to the task at hand, such as configuring a car computer. If the screen looked like it did the last time they shut it off, it at least would give the illusion of being on and would be a little less annoying. In the book The Humane Interface, Jef Raskin, father of the Macintosh computer, asserts that there are no real technical reasons why a computer should take any boot time at all.
Reducing boot times can be done by brute force or through finesse. The solutions we have found or been told about can be broken down into three basic categories:
Reduce the boot time to its theoretical minimum.
Distract the user while the computer is booting up.
Anticipate the user, and boot before the user has a chance to wait.
Long ago, I saw a comment on the Linux BIOS page that EPIA-M support existed but was unstable. That fact had not changed since July 2003. I e-mailed the maintainer of that branch, and he said "The fastest time that I have is 5s with some optimization of [the] init script."
Wow, five seconds. We wanted to get that.
Initially, I tried to find a commercial vendor to replace the BIOS. General Software was kind enough to send me a sample of its BIOS replacement for the EPIA-M, but the company did not yet have a similar version for the EPIA M2, another motherboard we use in our products. Also, the entry price was surprisingly consistent among the several vendors I asked--around $10,000 dollars. Recouping the effort of making a custom BIOS requires this sort of pricing, I suppose. Part of the problem was these BIOS implementations did all sorts of great stuff, and what we were in search of was a BIOS that did almost nothing.
So we set out to see how much time we could recapture by throwing away the BIOS and booting Linux ourselves. The idea was to throw away all the useless BIOS functionality, show video at the boot loader (that is, GRUB splash screens) and start playing audio as early as we possibly could in the boot sequence.
Our Linux engineer, James, was sent off with a pair of Epia-M boards, a bootable 256MB Flash drive, the first two boot articles and a pointer to the LinuxBIOS site. I initially thought that we could get enough Linux into the BIOS chip itself to do something useful, such as squeak and Flash the monitor. That would have satisfied our light and sound objective, but James discovered that we did not have enough space without going to the Flash drive. So the new goal was to reduce the BIOS boot as much as possible and then launch a stripped ZipSlack distribution on the compact Flash IDE drive.
The main benefit of LinuxBIOS for our application also was the main drawback. The primary development focuses on making clusters of supercomputers that need to be able to reboot quickly. These clusters were designed for network booting, however; we wanted to boot off a disk or a Flash device.
The EPIA-M boards, although quite popular in the hardware hacking scene are not popular as clustered supercomputing nodes. The EPIA is not the fastest motherboard, running between 600MHz and 1GHz with a comparatively feeble CPU.
As the LinuxBIOS project has progressed, some of the code to do things other than bootstrap supercomputer nodes has fallen out of sync with the documentation. Downloading and reading most of the LinuxBIOS mailing list archives helped to reconcile the discrepancies. For instance, for a while it seemed as though the Etherboot modules would be adaptable to booting off of other devices, such as the compact flash/IDE adapter combo we were using. Eventually, James switched to using the FILO bootloader.
The hardware aspects of flashing a BIOS were the most adventurous. The first few compiles of a new BIOS image naturally produce a machine unable to boot. The essential tools for BIOS programming work is two of the same motherboard and the correct chip tweezers.
Restoring a fried BIOS seems wrong when you're doing it. When you fry your BIOS in computer A, you boot computer B with the fixed BIOS and then carefully extract its chip with the computer still running. Then, you put the other BIOS chip in its place and burn it with a backup of the original BIOS. Then, you can put the chip back in the other computer and continue on your way.
Getting a usable BIOS was no small feat. Etherboot would fail in many charming ways, for any reason, such as not finding the Ethernet card, not finding the kernel on the disk and so on. After a bad flash, either there would be nothing at reboot or LinuxBIOS would fail to load the BIOS itself or FILO. After a dozen or so failed attempts, a bootable BIOS finally was flashed, and ZipSlack was booted successfully off the CF card.
The following several sections, written by James, detail what was discovered along the way.
Power down. With a Sharpie marker, mark your good BIOS with a G and the experimental BIOS chip with an X so you can tell them apart. Grab the chip tweezers tool. If there is enough clearance, put the hooks through the slots provided at the corners of the socket, press the tool against the socket and squeeze. If there isn't enough clearance to get both hooks in, unfold the tool, place one hook under one corner, press the tool against the socket and squeeze gently until the chip starts to come up. Then, move on to the other corner, then back to the first and so on. You want the chip to come out straight or it bends up the contacts.
Press the good BIOS in--make sure it is oriented correctly. It is possible to stick it in upside-down, which fries the chip. You don't have to push the chip all the way into the bottom of the socket.
Boot. You want to make sure your distro or Flash or what-not boots before you flash the BIOS.
As soon as it starts booting, you can pull the good BIOS, as it's been copied into shadow RAM. Pull it out as described above, being careful not to short out the board or break anything.
Push in the experimental BIOS.
Grab your fixed BIOS, however you choose to do that, and burn it with flash_rom from the FreeBIOS flash_and_burn directory. I found I often had to burn twice or, at least, had to wake the chip up by running flash_rom with no arguments to make it do the chip-ID stuff; I don't know why.
Reboot and see if it works.
The steps go in this order: power-up loads FreeBIOS, FreeBIOS loads a bootloader (FILO), the bootloader loads the kernel and so on. First, then, you have to build the bootloader. The FreeBIOS people like to use Etherboot, but Etherboot currently is broken as far as booting from disk goes. So, FILO is the thing to use. FILO is like GRUB, but it has no need for the BIOS.
Unpack FILO, switch to the FILO directory and run make. This creates a default config file. Edit config, then run make again to create filo.elf, which is what we use for the LinuxBIOS payload. Here's my FILO config.
Next, we need to add a video BIOS. Actually, the video BIOS on the Epia-M is pretty useless (more on this later), but it serves to let X know that we have built-in video, so in it goes.
Because the video BIOS is copyrighted by VIA and cannot be distributed independently, you have to grab it from your original BIOS. Do so in one of two ways. Either boot Linux and enter:
dd if=/dev/mem of=vgabios.bin skip=1536 count=128
or extract it from the BIOS distribution image, if you can find the offset. Cat the two together, like so:
cat vgabios.bin filo.elf > vfilo.elf
Therefore, vfilo.elf is our payload--a bootloader plus a video BIOS.
The traditional LinuxBIOS now is called FreeBIOS, as it can load things other than Linux. A new rewrite, called FreeBIOS2, doesn't appear to be ready yet. I therefore used FreeBIOS, which is available only from CVS. To get it, type:
cvs -d:pserver:email@example.com:/cvsroot/freebios login cvs -z3 -d:pserver:firstname.lastname@example.org:/cvsroot/freebios co freebios
FreeBIOS builds and configures in a separate directory, so doing a CVS update doesn't hose your existing config files. So, in the same directory as where you have /linuxbios, make a build directory by issuing mkdir build. Then, create an epia-config file; here's our epia-config file.
Notice all the full paths in that file? LinuxBIOS likes full paths. To build the BIOS, do this:
python /home/jamesh/linuxbios/freebios/util/config/NLBConfig.py epia-config/home/jamesh/linuxbios/freebios/
where /home/jamesh/linuxbios/freebios is where I put FreeBIOS. Then run make, which should produce a file called romimage. This is what you burn to your BIOS chip.
You also are going to need some utilities from the FreeBIOS folks. cd into freebios/util/mkelfImage and issue
./configure && make && make install
to get mkelfImage, which should end up in /usr/local/sbin. Then, cd into ../flash_and_burn and do make to get flash_rom. This is what you will flash your ROM with.
On a side note, some other tools are available to produce ELF kernels--don't use them. Most of them are broken, including the one from the Etherboot project that is mentioned in the HOWTOs.
The Epia patches include one called epia-fbdev, which leads to this encouragement from the LinuxBIOS mailing list:
El Jueves, 13 de Mayo de 2004 18:46, John Laur escribi?: > If you have VGA correctly working with LinuxBIOS (Ie you see the > linuxbios logo on startup), then you have VGA working as best it does > with LinuxBIOS. > > Then you can worry about the kernel. VESA Framebuffer does not work with > EPIA-M as the VGA bios doesn't initialize the video card up to where > vesafb can take over. You have to use viafb (part of the -epia patchset) > to get video on the console. Make sure you have it compiled into the > kernel and not as a module. Make sure vesafb is NOT compiled in either, > otherwise you'll need some commandline trickery to make it work. Also, > viafb doesn't work with bootsplash in silent mode on kernel 2.4 if you > are using that. I have not tested with kernel 2.6. > > John
However, all the flavors of epia-fbdev I tried hung the machine or produced scrambled text.
Anyway, one should build a kernel in the normal way. I unpacked the kernel, configured it and issue
make dep && make bzImage && make modules
Then, to install the modules somewhere so I could copy them to the Flash, I entered:
INSTALL_MOD_PATH=/home2/jamesh/linux-fast/ make modules_install
That command created /home2/jamesh/linux-fast/lib/modules/2.4.26-fast/. Next, make an elf image out of the kernel. This lets you put the command-line arguments in the kernel file itself and allows the bootloader to determine how big the kernel is:
/usr/local/sbin/mkelfImage --kernel=linux-2.4.26-fast/arch/i386/boot/bzImage --output=bzImage-2.4.26-fast.elf --command-line="root=/dev/hda4 console=tty0 console=ttyS0,115200n8 rw"
Note that I set root=/dev/hda4. I also set the FILO like so:
AUTOBOOT_FILE = "hda4:/bzImage"
Now, I only needed to put the kernel in /dev/hda4 in the root directory--easy. Next, copy the romimage and flash_and_burn files to your Epia box and run flash_and_burn by itself to see if it recognizes your Flash chip. Typing flash_and_burn romimage burns it.
Now, hook a terminal up to the serial port, set to 115200-n-8-1, and boot. In 20 seconds you should hear Dexter, assuming you are using my ZipSlack image. As you can see, X works but the mouse does not--I don't know why. Type startx to start X.
I've created two disk images. One is UMSDOS, which I unzip into a MS-DOS filesystem on partition 4 of a disk. The other is a tar file that should work on an ext2 or ext3 filesystem.
The full log of our successful boot with fine-grained timings is here, and the following guided tour takes you through the highlights.
I wrote a program to dump the console out from the serial port with a timestamp. Here are some interesting things to note:
It takes .334 seconds to get from power-on to the BIOS.
> 3.492 Copying LinuxBIOS to ram.
Three and a half seconds, and now it's copying the BIOS to shadow. I assume that time was spent finding shadow itself.
> 6.292 Jumping to LinuxBIOS.
Here's a major timewaster: reading from the Flash BIOS chip is slow.
> 6.616 POST: 0x75 > ... > 7.076 POST: 0x95
I have no idea what this really means.
> 7.153 found VGA: vid=1106, did=3122 > ... > 9.533 biosint: Unsupport int #0xcd
This attempts to run the video BIOS I installed. 2.38 seconds can be saved by removing the dir src/bioscall line in epia-config in the FreeBIOS build directory. The only build I have at the moment that skips this step also doesn't send anything to the serial port, so I don't have a log of it.
> 9.574 Welcome to elfboot, the open sourced starter.
We're not out of LinuxBIOS yet.
> 9.722 FILO version 0.4.2
We've finally moved out of LinuxBIOS and into the FILO bootloader.
> 9.792 No sound device found
Yes, that message is in FILO. If you have a sound device that FILO recognizes, it would play a sound at this point. However, I couldn't get it to find the the VIA sound device, not that I tried very hard. But if you were using a mobo with a standalone video card and SB-compatible sound, we would have picture and sound at just under ten seconds.
> 10.013 Loading Linux version ... > 12.242 Jumping to entry point...
Hmm, over two seconds to get the kernel off the Flash and into RAM.
> 13.252 Linux version 2.4.26-fast
This is the actual start of the Linux boot.
> 19.811 Playing Sound
At this point sound is playing out the soundcard--19.811 seconds, or 17.431 if you skip the video BIOS call. Of this, only 6.559 seconds actually is booting Linux.
We did not get the five-second boot we were hoping for. Several of the reasons were specific to the EPIA board we used. The on-board EPIA video presented a problem, too, because its own video BIOS was part of the computer BIOS. Had it been a separate video card, its initialization could have been skipped altogether, leaving it for X to do when it launched. Similarly, if we had been using a more conventional sound card, such as an old SoundBlaster, we could have had sound much earlier in the boot process, around ten seconds.
The exercise did illustrate many useful things, however. It gave us a precise breakdown of where time was being lost, and it did give us a lower threshold--around ten seconds--in which we know we can get sound and video out to the user, given the right motherboard and hardware. And this is from a cold boot, pressing the power switch.
Although we never did get the video to display, even after X was booted, good news on this front came as this article was being completed. An implementation of FreeBIOS for the EPIA-M2 board has been completed by Nick Barker, and it seems to solve not only the video problems but a longstanding difficulty of booting directly from the CF slot on these boards. With further work, we could get our applications down to this new theoretical ten-second minimum.
But there are other ways to reduce the boot time for an in-car computer. To conclude, I'll list a few of the options we haven't covered yet.
Obvious methods include brute force--using faster processors and hardware.
Hibernation, saving the memory state to hard disk and then restoring it on boot, is an excellent tool. But as memory sizes grow, the speed of this solution decreases.
Emulating the PalmPilot: install a second battery in the car, using a voltage isolator, and run the computer all the time. It will sleep and boot up in seconds when you wake it up.
Schedule your computer. Using the BIOS wake-up feature in some computers, set it to wake up a few minutes before you normally go to your car in the morning. By doing so, it will be booted fully before you even get to your car.
One reader of the last article suggested making the computer start booting not when you turn the key but as soon as you open the car door. An improvement to this technique is to boot as soon as the user unlocks the door with the remote keyless entry.
As general-purpose computers find their way into cars in greater and greater numbers, people will keep hammering away at boot times.