Adding custom syscall to Linux

Introduction

System calls (abbreviated syscalls) are the gateway for userspace applications to kernel. They are used to ask the kernel to provide a service (like reading or writing a file content, ..., etc).

An exhaustive list of syscalls is available at : http://man7.org/linux/man-pages/man2/syscalls.2.html.

Monitoring system calls from userspace

One can use strace to track different system calls make by an application toward the kernel. The following shows an example of syscalls made by ls:

We can clearly see the different system calls (open, close, mmap, fstat, ..., etc).

Custom system call

Setting up Linux sources

Getting Linux kernel sources

Let's download the Linux sources from https://www.kernel.org/ as shown below :

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.0.2.tar.xz

Note : At the time of writing this post, Linux 5.0.2 is the latest version.

Decompress the sources

One the sources have been downloaded, We need to decompress and copy them into /usr/src as follow:

$ sudo tar -xvf linux-5.0.2.tar.xz -C /usr/src/

Install required packages

The following packages are required to successfully compile a linux kernel :

$ sudo apt-get install build-essential flex libncurses5-dev bison libssl-dev libelf-dev

Adding a syscall

Declaring the syscall prototype

System call prototypes are declared in : linux-sources/include/linux/syscalls.h.
In our case; We need to add our custom syscall prototype into /usr/src/linux-5.0.2/include/linux/syscalls.h. One can follow the steps given below :

$ cd /usr/src/linux-5.0.2
$ sudo nano include/linux/syscalls.h

Then add you syscall prototype at the end of the file before #endif.

Syscall implementation

• Create a folder at the root of the sources, let's call it custom_syscall.

  $ cd /usr/src/linux-5.0.2
  $ sudo mkdir custom_syscall
  $ cd custom_syscall
  

• Add the syscall implementation source file :

  $ sudo nano custom_syscall.c
  

  

  and provide a Makefile :

  $ sudo nano Makefile
  

  
• Add you syscall to the Linux Main Makefile

  $ cd /usr/src/linux-5.0.2
  $ sudo nano Makefile
  

  
• Adding syscall to system call table

  $ cd /usr/src/linux-5.0.2/arch/x86/entry/syscalls/
  $ sudo nano syscall_64.tbl
  

  

Compiling Linux kernel sources

At this step, The linux source directory should look like the following:

• Create a Linux image : one can call make command passing it the number of processors*2 (to speed up compiling process by using multiple processors).
  For example, I have 2 processors on my machine, so I have to call make as follow:

  $ sudo make -j4
  

  Where 4 is the number of processors on my machine * 2.
• Compile modules :

  $ sudo make modules_install install
  

  The above command generates a bench of files like : vmlinux-5.0.2 and system.map-5.0.2. One may take a look at the boot folder :
  
• Check you kernel version before reboot :
  
• Reboot your system :

  $ sudo shutdown -r
  

• Check your new kernel version :
  

Testing system call

• Write a test program : The function syscall performs a generic system call. It's prototype is defined as follow:

  long int syscall (long int sysno, ...)
  

  The extras arguments are the arguments of the syscall, however; our system call does not have any argument. One may write a test code as shown below :

  #include <stdio.h>
  #include <linux/kernel.h>
  #include <stdlib.h>
  #include <sys/syscall.h>
  #include <unistd.h>
  
  int main(int argc, char *argv[]){
      // 335 is the syscall number defined in syscall_64.tbl
   long int sysReturnCode = syscall(335);
   printf("Syscall returned value :%ld\n", sysReturnCode); 
   return EXIT_SUCCESS; 
  }
  

• Read syscall output : Use the dmesg command which should yield :
  

Congratulations! Now, try to create more enhanced systemcall (and put them in the comments below).