The Linux Trace Toolkit
As recent Linux history has shown (Mindcraft, anyone?), performance is not only good publicity, it's important. Yet current means of measuring performance offer only global statistics about the whole system or very precise data about an isolated application. Moreover, these often fail in helping the programmer or the system administrator to isolate a performance bottleneck resulting from the interaction of complex internetworking applications, which are more and more common. The Linux Trace Toolkit (LTT) addresses these issues and provides users with a unique view of the system's behavior with minimal performance overhead (< 2.5%).
In order to be extendable and accomplish its task without hindering system performance, LTT is designed to be as modular as possible. In fact, it would be wrong to call it a “tool” since it is composed of many pieces that, grouped together, fulfill the desired function. This toolkit is implemented in four parts. First, there is a Linux kernel that enables events to be logged. Second, a Linux kernel module takes care of storing the events into its buffer and then signals the trace daemon when it reaches a certain threshold. The latter then reads the data from the module, which is visible from user space as a character device. Last, but certainly not least, the data decoder takes the raw trace data and puts it in a human-readable format while performing some basic and more advanced analysis. This decoder, as will be discussed further, serves as the toolkit's graphic and command-line front end.
The LTT tar.gz archive can be found at http://www.opersys.com/LTT/ and contains the following items:
Copying: the GNU GPL License
Help: LTT's help files in an HTML-browsable format
TraceDaemon: the directory containing the trace daemon
TraceToolkit: the directory containing the trace toolkit front end
patch-ltt-kernelversion-yymmdd: the kernel patch of yymmdd kernelversion
trace: a script to start the trace daemon
tracecpuid: another script to start the trace daemon
tracedump: a script to dump the content of trace
traceanalyze: a script to analyze a trace
traceview: a script to view a trace in graphical form
The scripts are there to speed up the tools' most common usages, but the tools can be summoned directly without any script.
To install LTT, simply follow the instructions that come with the LTT package. The first and hardest step is patching the kernel. Once this is done, configure the kernel and compile it. Note that there is an option for compiling with or without the tracing code. When compiled without, the resulting kernel operates as if you hadn't applied any patch to it. Next, compile the trace daemon and the trace toolkit graphic front end and put them in your favorite directory (/usr/bin or /usr/local/bin for example). Reboot with the LTT patched kernel, and you're ready to go.
To demonstrate the toolkit's operation, we traced 10 seconds' worth of system operation. During those 10 seconds, two commands were issued: dir on a directory not accessed since system boot (i.e., not present in the dcache) and bzip2 on a 10MB file. The system was booted in single-user mode (in order to have as few applications running as possible, and therefore isolate the operation of the observed applications) using the modified kernel. Note that no events are recorded by the kernel module until the trace daemon has issued the start command to it using the ioctl system call. The following command was issued to start the tracing:
trace 10 out140
trace is a script that takes two arguments: the number of seconds the trace should last and the base name for the output file. Two files are produced: out140.trace and out140.proc. The former holds the data recorded by the kernel module, and the latter, the content of /proc when the trace started. Using these two files, we know what the system looked like before we started tracing it and what happened during the trace. Hence, we can reconstruct the system's behavior.
Note that the trace daemon accepts many command-line options used to configure the kernel trace module. For instance, one can specify the events to be traced and the desired level of details. One can also specify whether CPU IDs should be recorded for SMP machines. Since LTT fetches the calling address for system calls, you can specify at which calling depth this address should be fetched or which address range it is a part of.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SUSE LLC's SUSE Manager
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- Managing Linux Using Puppet
- My +1 Sword of Productivity
- Non-Linux FOSS: Caffeine!
- Doing for User Space What We Did for Kernel Space
- Google's SwiftShader Released
- SuperTuxKart 0.9.2 Released
- Rogue Wave Software's Zend Server
- Parsing an RSS News Feed with a Bash Script
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide