An introductory tutorial on how to develop and test Linux kernel
modules
Manolis Marazakis
May 10, 2019
In this tutorial …
• Familiarization with Linux kernel modules
– Two examples
• Learn about ways to explore the operation of a live Linux system
• Introduction to PCI device drivers
Introduction to Linux Kernel Modules
Introduction to Linux Kernel Module Programming (aka: I can write a driver me!)
Linux Kernel module : code that can be loaded/unloaded without system reboot- Extensions of kernel functionality, in the form of a “special” object file- Loadable vs. Built-in- Device driver allows kernel to access hardware connected to the system
The kernel & its modules essentially are a single program (& single namespace)
Credit for title: “Brainiac: Science Abuse”[ https://en.wikipedia.org/wiki/Brainiac:_Science_Abuse ]
Introduction to Linux Kernel Modules
Computer systems are built on levels of abstraction
• Different perspectives on what a “machine” is– OS ISA: Instruction Set
Architecture• h/w – s/w interface
– Compiler ABI: Application Binary Interface• User ISA + OS calls• Calling conventions
– Application API: Application Programming Interface• User ISA + Library calls
Introduction to Linux Kernel Modules
ISA
ABIAPI
The OS as a resource manager
• Operating system := software that controls and manages the hardware and system resources of a computer
– Abstraction and Isolation
– Virtualization of system platform resources: processes, memory, filesystem, devices
Introduction to Linux Kernel Modules
+ Concurrency, +Persistence …
Kernel space
• The executing code has complete and unrestricted access to the underlying hardware.
– Can execute any CPU instruction and reference any memory address.
– Intended for the lowest-level, most trusted functions of the OS.
– Crashes in kernel mode are catastrophic; they will halt the entire PC.
Introduction to Linux Kernel Modules
User space
• The executing code has no ability to directlyaccess hardware or reference memory.
– Code running in user mode must delegate to system APIs to access hardware or memory.
– Due to the protection afforded by this sort of isolation, crashes in user mode are always recoverable.
Introduction to Linux Kernel Modules
Kernel vs User: enforced by CPU HWIf code executing in User modeattempts to do something outside itspurview– e.g. accessing a privilegedCPU instruction or modifyingmemory that it has no access to -- atrappable exception is thrown Instead of the entire systemcrashing, only that particularapplication crashes.
Introduction to Linux Kernel Modules
Protection rings (x86)
Linux: Kernel- vs. User- Space
Introduction to Linux Kernel Modules
system call: - A function stub - entry point for requesting OS services.
Embrace change (with kernel modules)
Introduction to Linux Kernel Modules
View list of modules: /proc/modules (also: lsmod utility)
“Hello World” example/**
* lkm0.c : "null" Linux kernel module
* FOSSCOMM 2018 @ U. Crete, Heraklion, Greece
*/
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
MODULE_LICENSE("GPL");
MODULE_AUTHOR("FOSSCOM18");
MODULE_DESCRIPTION("A Linux kernel module for tutorial use");
MODULE_VERSION("0.00");
static int __init lkm_example_init(void) {
printk(KERN_INFO "[%s] Module initialed. \n", __FUNCTION__);
return 0;
}
static void __exit lkm_example_exit(void) {
printk(KERN_INFO "[%s] Module removed. \n", __FUNCTION__);
}
module_init(lkm_example_init);
module_exit(lkm_example_exit);Introduction to Linux Kernel Modules
Modules are built by the kbuild system
obj-m += lkm0.o
KERNEL = $(shell uname -r)
all:
make -C /lib/modules/$(KERNEL)/build M=$(PWD) modules
clean:
make -C /lib/modules/$(KERNEL)/build M=$(PWD) clean
To enable building kernel modules: apt-get install linux-headers-$(uname -r)
Introduction to Linux Kernel Modules
Exploring our first module: lkm0.ko
# sudo insmod ./lkm0.ko# cat /proc/modules# cat /proc/devices# lsmod# dmesg view kernel log# sudo rmmod lkm0
Introduction to Linux Kernel Modules
sysfs
• sysfs: a RAM-based filesystem, for exporting to user-space kernel data structures, their attributes, and the linkages between them – tied inherently to the kobject infrastructure
– mount -t sysfs sysfs /sys
• [ https://www.kernel.org/doc/Documentation/filesystems/sysfs.txt ]
Introduction to Linux Kernel Modules
A character driver (lkm1.c)#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <asm/uaccess.h>
#include <linux/device.h>
#include <linux/sysfs.h>
#include <linux/slab.h>
….
static struct cdev *fosscomm18_cdev;
static struct device *fosscomm18_device;
static struct class *fosscomm18_class;
static dev_t fosscomm18_dev;
static char *fosscomm18_buffer; /* dynamic allocated */
static int fosscomm18_buffer_end = -1;
static atomic_t fosscomm18_buffer_start; /* may be changed by sysfs attr */
Introduction to Linux Kernel Modules
A character driver: readstatic struct file_operations fops = {
.owner = THIS_MODULE,
.read = dev_read,
.write = dev_write,
.open = dev_open,
.release = dev_release};static ssize_t dev_read(struct file *fp, char *buf, size_t len, loff_t *off) {
unsigned long rval; size_t copied; if (len > (fosscomm18_buffer_end - *off))
len = fosscomm18_buffer_end - *off; rval = raw_copy_to_user(buf, fosscomm18_buffer + *off, len); if (rval < 0) return -EFAULT; copied = len - rval; *off += copied; return copied;
}
Introduction to Linux Kernel Modules
A character driver: writestatic ssize_t dev_write(struct file *fp, const char *buf, size_t len, loff_t *off)
{
unsigned long rval; size_t copied;
printk(KERN_DEBUG DEVICE_NAME
" dev_write(fp, buf, len = %zu, off = %d\n", len, (int)*off);
if (len > fosscomm18_buffer_end - *off)
len = fosscomm18_buffer_end - *off;
rval = raw_copy_from_user(
fosscomm18_buffer + atomic_read(&fosscomm18_buffer_start), buf, len);
if (rval < 0) {
printk(KERN_DEBUG DEVICE_NAME " copy_from_user() failed\n");
return -EFAULT;
}
copied = len - rval;
*off += copied;
return copied;
}
Introduction to Linux Kernel Modules
A character driver: initialization
Introduction to Linux Kernel Modules
fosscomm18_buffer = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!fosscomm18_buffer) return -ENOMEM;
fosscomm18_buffer_end = PAGE_SIZE;
rval = alloc_chrdev_region(&fosscomm18_dev, 1, 1, DEVICE_NAME);
fosscomm18_cdev = cdev_alloc();
rval = cdev_add(fosscomm18_cdev, fosscomm18_dev, 1);
fosscomm18_class = class_create(THIS_MODULE, CLASS_NAME);
fosscomm18_device = device_create(fosscomm18_class, NULL, fosscomm18_dev, NULL, DEVICE_NAME);
rval = device_create_file(fosscomm18_device, &dev_attr_buffer_start);
rval = device_create_file(fosscomm18_device, &dev_attr_buffer_end);
A character driver: sysfsstatic ssize_t buffer_start_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n“, atomic_read(&fosscomm18_buffer_start));
}
static ssize_t buffer_end_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n", fosscomm18_buffer_end);
}
Introduction to Linux Kernel Modules
Exploring the character driver module
Introduction to Linux Kernel Modules
Curious enough to explore further ?• Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau, “Operating Systems: “Three
easy pieces” [ http://pages.cs.wisc.edu/~remzi/OSTEP/ ]
• Write a Real Linux Driver [ webinar by Greg Kroah-Hartman ]
– https://training.linuxfoundation.org/resources/webinars/write-a-real-linux-driver/
• Linux kernel development tutorial
– Video of Greg Kroah-Hartman’s talk at Git Merge 2016https://www.youtube.com/watch?v=vyenmLqJQjs
• Linux kernel driver interface
– https://www.kernel.org/doc/html/v4.15/process/stable-api-nonsense.html
• LDD3 book : https://lwn.net/Kernel/LDD3/
– Warning: dated (but explains core principles)
• LXR (cross-referenced source code)
– https://elixir.bootlin.com/linux/latest/source
• Tutorial on Linux PCI Drivers– http://freeelectrons.com/docs/pcidrivers
Introduction to Linux Kernel Modules
What’s going on ???
• The Linux kernel has three primary mechanisms for kernel tracing and profiling:– tracepoints – a mechanism that works over static instrumented
code– kprobes – a dynamic tracing mechanism used to interrupt a
kernel code at any point, call its own handler, and return after all of the necessary operations have been completed
– perf_events – an interface for accessing the PMU (Performance Monitoring Unit)
• cat /sys/kernel/debug/tracing/available_tracers– hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt
wakeup function nop
• Front-ends for ftrace: trace-cmd, kernelshark
Introduction to Linux Kernel Modules
# trace-cmd record -e ext4 ls# trace-cmd report
Introduction to Linux Kernel Modules
Try also: # trace-cmd record -p function_graph –P <PID> (e.g. PID of sshd)# trace-cmd list –f# cat /sys/kernel/debug/tracing/available_events
ftrace (via trace-cmd)
• ftrace tracing framework
– https://www.kernel.org/doc/Documentation/trace/ftrace.txt
• trace-cmd: A front-end for Ftrace
– https://lwn.net/Articles/410200/
• Getting started with ftrace (Julia Evans’s blog)
– https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/
Introduction to Linux Kernel Modules
perf-tools
Introduction to Linux Kernel Modules
PCI Devices in Linux (1)
Introduction to Linux Kernel Modules
/sys/devices/pci0000:00/0000:00:1e.0/0000:02:01.2
PCI Devices in Linux (2)
• Configuration space (256 bytes), per device– View via: lspci –x
• Standardized format (first 64 bytes)– Offset 0: Vendor ID, Offset: 2: Device ID
– Offset 10: Class ID (e.g. NIC, video adapter, bridge)
– Offsets 16-39: BAR [0-5]• Base Address Registers
– Offset 44: Subvendor ID | Offset 46: Subdevice ID
– Rest of config. Space is up to the manufacturer
Introduction to Linux Kernel Modules
Linux PCI Drivers (1)
• Device resources (I/O addresses, IRQ lines) are assigned at boot time
• PCI drivers “simply” have to read the corresponding configurations
• PCI config. space is Little-Endian.
Introduction to Linux Kernel Modules
Linux PCI Drivers (2)
• Declaration of driver hooks & supported devicesstatic struct pci_device_id pci_ids[] = {
{ PCI_DEVICE(QEMU_VENDOR_ID, EDU_DEVICE_ID), },{ 0, }
};MODULE_DEVICE_TABLE(pci, pci_ids);
static struct pci_driver pci_driver = {.name = “edu_pci",.id_table = pci_ids,.probe = pci_probe, /* called by generic PCI subsystem for matching devices */.remove = pci_remove/* could also provide hooks for PM: suspend/resume */
};
Introduction to Linux Kernel Modules
Linux PCI Drivers (3)
static int __init myinit(void) {
if (pci_register_driver(&pci_driver) < 0) { return 1; }
return 0;
}
static void __exit myexit(void) {
pci_unregister_driver(&pci_driver);
}
module_init(myinit);
module_exit(myexit);
Introduction to Linux Kernel Modules
Linux PCI Drivers (4)
static int pci_probe(struct pci_dev *dev,
const struct pci_device_id *id)
static void pci_remove(struct pci_dev *dev)
Introduction to Linux Kernel Modules
Warning – things might break … but you can fix them!
Introduction to Linux Kernel Modules
• ctags, find, grep, vim/emacs, … • printk() calls (conditional), sysfs, /proc, debugfs• _many_ kernel configuration features (via
Kconfig settings)• gdb vmlinux /proc/kcore• nm, objdump, /proc/kallsyms• … and you’ll grow to love virtual machines :-D
Don’t panic, everything is under control
Introduction to Linux Kernel Modules
Thank you for attention!