Implementing Linux System Calls

How to create and install a system call in Linux and install an interrupt for controlling the serial port.

This article is based on my experiences in creating and installing a system call in Linux and how to install one interrupt vector to control the serial port. In one way, this is a mini-HOWTO about these two topics.

What is a System Call?

A system call (or system request) is a call to the kernel in order to execute a specific function that controls a device or executes a privileged instruction. The way system calls are handled is up to the processor. Usually, a call to the kernel is due to an interrupt or exception; in the call, there is a request to execute something special. For example, the serial port may be programmed to assert an interrupt when some character has arrived, instead of polling for it. This way, the processor can be used by other processes and service the serial port only when it is required.

The internal operation between an interrupt request and its servicing involve several CPU registers and memory segments. Briefly, a device raises an interrupt by asserting an interrupt request line on the Peripheral Interrupt Controller (PIC) which informs the CPU by setting the interrupt request pin. After each instruction, the CPU checks this pin. If it is enabled, it gets the ID from the data bus, which points to the Interrupt Descriptor Table (IDT), where a number of task, interrupt and gate descriptors are stored. The descriptor contains a selector to the Global Descriptor Table (GDT) which contains the base address to a memory segment in which the Interrupt Service Routine (ISR) resides.

Note that the CPU has suspended the process it was executing, so it has to save some information to be able to resume the process after the interrupt has been serviced—this is a context switching. Several files are involved in this process; most can be found in the linux/arch/i386/kernel/ directory. One is entry.S, an entry point to all system calls which initializes the treatment of exceptions. Another is irq.c, which contains the functions to deal with interrupts. The linux/arch/i386/boot/setup.S file initializes the GDT, installs virtual memory, etc. There are many connections between files ending in .h and .c. You can check irq.c to see how many includes are there to get macros definitions, such as cli(), which clears interrupts in linux/include/asm-i386/system.h.

To follow the definition path of any function, type at your command prompt:

grep cli 'find / -name '*.[ch]'

This will search all files with extension c and h in the root directory for the word cli. Also, you can issue the command man 2 intro to see something about system calls.

Implementation of System Calls

There are several ways to create, install and execute a system call. The best is the one that isn't concerned with low-level details like context switching and doesn't code any routines in assembly language. This can be done through the use of the _syscallN macro in the linux/include/asm/unistd.h directory; it is expanded in assembly, but the operating system takes care of details. It uses the int 0x80 to transfer execution control to the kernel. One possible problem is this macro can expand to an existing function, so care must be taken; otherwise, you will overwrite the existing function.

In order to implement your own system calls, you should have the Linux kernel source code (first make a backup) to use as the working copy. As superuser (root), create in your home directory an entire tree copy of /usr/src/linux, as you will not have the chance to do so again. The files we will use are in somewhere/linux/.

Now you must choose a name for every function you are planning to implement. You can check the existing ones in your source tree at linux/arch/i386/kernel/entry.S and linux/include/asm/unistd.h. In entry.S, they are at the end, and in unistd.h, at the beginning. Checking these files will also help you get an idea of how to create a prototype of a system call. While checking, you will see that each call is associated with one number. This number is passed in the %eax processor register indicating the number of arguments, and each argument of the system call (a function) is passed in %ebx, %ecx, %edx, %esi or %edi--up to five arguments on Intel platforms. The macro definitions corresponding to each _syscallN, depending on the value of N, can be found in unistd.h. More on the internal workings can be found in various files under linux/arch/i386/, because we will leave the “dirty work” to the operating system.

Now let's see how to implement a new system call using the syscallN macro in the simplest possible way. Let's make a system call sysSum, which accepts two integer arguments and returns the sum of the two. Also, it uses printk, which is similar to printf except that it works on the kernel level, so we will see when our function is called.

To do this, edit a randomly selected file (for example, the file linux/ipc/sem.c), and at the end, add the following lines:

asmlinkage int sysSum(int a, int b)
        printk("calling sysSum\n");
        return a+b;

Then edit unistd.h and add

#define __NR_sysSum     171
171 is the next in numerical order. In entry.S near the end, add
.long SYMBOL_NAME(sysSum)
Finally, increment by one the number that is the last line:
.space (NR_syscall-172)*4
If you don't match number and name in both files, you will get an “undefined reference to sysSum” error message. If you have a working kernel, you have to be careful only about incrementing the numbers by one and correctly writing your function name. At this point, you have added your system call; now you should get the new kernel with it. To recompile the kernel, take the following sequential steps:
#make config
#make dep
#make clean
#make zImage
#cat ~/linux/arch/i386/boot/zImage >/dev/fd0
Step 1 creates the basic kernel configuration; you can skip it next time if no hardware changes are made. Step 2 checks that any dependency between files is correct. Step 3 cleans any compilation intermediate file (object files, etc.). The last two create a compressed kernel image and copy it to floppy, so we can try our new kernel and keep the original one untouched.

Reboot using this newly created kernel to invoke the system call from a user program: simply insert the floppy disk on the drive and reboot. This simple program tests the newly created system call:

#include <linux/unistd.h>
_syscall2(int, sysSum, int,a,int,b)
printf("the sum of 4+3 is %d\n",sysSum(4,3));

The include line indicates where the _syscall definition is located. The next line says our system call has a return type of int and two arguments of type int. To compile, use the command

gcc -I ~/linux/include
to instruct the compiler to use our include file. After execution, you will see messages: first the one from sysSum, then the one from the test program.

The functions we will implement will be the basic ones needed to control the serial port using interrupts on character reception. The serial ports can't be accessed by a common user. In Linux, the functions inb(port) and outb(byte, port) exist to receive and send one byte; inw and outw do the same on two-byte data. In order to use them, you have to gain the rights by using the iopl or ioperm functions, which must be invoked as super user and will give the common user application access to the I/O ports.



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Anonymous's picture

There is an updated list of Linux system calls here

Implent by S/m call

jamsheeralimca's picture

The Concept of Implement System Calls in C program is Realy helping to Development of My Small Project.Lab Section. So I am realy Thanks.............................


Anonymous's picture

really its good . every step is mentioned in very clear manner.

Great Article

kamal Maiti's picture

It's a great article that I ever have seen. Thanks a lot for this. It has cleared some confusion regarding the system call and their interface.


Anonymous's picture

its the nice artical.those who want to do the work in the area of system programming should read this one.

Nice Article

Anonymous's picture

This is really a nice article. Many important things are eloberated nicely here. All system programmers, linux explorers should read this article. :)

Free Dummies Books
Continuous Engineering


  • What continuous engineering is
  • How to continuously improve complex product designs
  • How to anticipate and respond to markets and clients
  • How to get the most out of your engineering resources

Get your free book now

Sponsored by IBM

Free Dummies Books
Service Virtualization

Learn to:

  • Define service virtualization
  • Select the most beneficial services to virtualize
  • Improve your traditional approach to testing
  • Deliver higher-quality software faster

Get your free book now

Sponsored by IBM