Reducing Boot Time in Embedded Linux Systems
Listing 1. initcalls Taking More than Nine Milliseconds to Complete
root@8548cds:~# dmesg | grep initcall | egrep '[0-9][0-9] msecs' initcall pty_init+0x0/0x43c returned 0 after 57 msecs initcall serial8250_init+0x0/0x138 returned 0 after 20 msecs initcall gfar_init+0x0/0x58 returned 0 after 60 msecs initcall cp_init+0x0/0x34 returned 0 after 16 msecs initcall ide_scan_pcibus+0x0/0x14c returned 0 after 4246 msecs initcall of_flash_init+0x0/0x34 returned 0 after 43 msecs initcall uhci_hcd_init+0x0/0x104 returned 0 after 445 msecs initcall ip_auto_config+0x0/0xefc returned 0 after 4597 msecs
Using initcall_debug in this scenario reveals that almost nine seconds could be saved by eliminating or deferring ide and IP auto configuration!
One of the more powerful tools for boot time measurement is Kernel Function Trace. KFT instructs the compiler to generate instrumentation for virtually all kernel function calls. When enabled and triggered, data is logged for each function call entry and exit, which allows you to identify functions that consume large amounts of time. To use this tool, you will need to apply the KFT patch to your kernel. Detailed instructions and links to KFT kernel patches can be found at elinux.org/Kernel_Function_Trace.
For each function call not specifically filtered out by your trace configuration, a line is generated that contains a timestamp at function entry, the function address, the address of the caller, process ID and a delta. The raw data is accessed by reading /proc/kft_data after the run has completed.
Several tools are available to post-process the raw data. The addr2sym converts the kernel virtual address in the raw data to symbolic addresses. It is simple to use:
# addr2sym <kft_data.raw -m System.map >kft_data.sym
Above, kft_data.raw is the raw data copied from /proc. System.map is produced by the kernel build and can be found in the top-level kernel directory, and of course, kft_data.sym is the output file.
The KFT dump (kd) utility can be found in the Linux kernel scripts directory after the KFT patch has been applied. Running kd on the raw data produces a statistical summary of the functions called. You can use kd to display the most time-consuming functions or functions with time greater than that specified in your configuration and several other useful filters. Listing 2 contains a partial listing (top ten) of the most time-consuming functions on a typical high-performance Power Architecture processor. The negative number associated with the schedule call is due to the fact that schedule changes context to another process—that is, it never exits in the traditional manner that a function call usually does.
Listing 2. Using kd
$ ./linux/scripts/kd -n 10 kft_data.sym Function Count Time Average Local ----------------------------- ----- -------- -------- -------- __schedule 5208 22050824 4234 22046510 schedule 1921 10828704 5637 -10478620 setup_arch 1 6021110 6021110 29 tsc_init 1 6021081 6021081 79 set_cyc2ns_scale 1 6021002 6021002 6021002 kobject_uevent 389 1659254 4265 813013 mem_init 2 1223745 611872 111906 wait_for_completion 395 1192559 3019 14685 free_all_bootmem 1 1109561 1109561 53 free_all_bootmem_core 1 1109508 1109508 74651
Improving Linux boot time is moving from the obscure corners of R&D labs to mainstream product development. Driven by competitive pressures in a wide variety of markets, system developers are devoting an increasing amount of effort to making sure their systems are ready to use when users want them. Always a hot topic, we are sure to see many more developments in the near future aimed at further reducing Linux system boot time.
Other tools are available to help reduce your system's boot time. Bootchart is a powerful tool useful for visualizing the post-kernel initialization processes. Details can be found at www.bootchart.org.
Readahead is designed to pre-fetch required boot files from disk so that when they are needed, they can be read from the buffer cache for faster boot. Readahead can be customized to read specific files in a given order. You can find more about readahead at https://fedorahosted.org/readahead.
For the ambitious, the Moblin distribution contains a host of optimizations, which taken together, aim to produce a five-second boot on a typical Netbook: moblin.org/projects/fast-boot.
Elinux.org (elinux.org/Boot_Time) maintains a very useful collection of data related to fast boot optimizations, including more analysis and profiling tools, links to other articles and much more.
Christopher Hallinan is the author of Embedded Linux Primer and a Field Applications Engineer for MontaVista Software, Inc. He has been engaged in Linux-related work and play since 2000. He currently resides in sunny southwest Florida.