IntroductionTo

Kernel Modules

- Dibyajyoti Ghosh

IntroductionWhat is kernelKernel Space and User SpaceKernel CategoriesLinux Kernel SourceBuild your own KernelKernel FilesKernel ModulesKernel Module Build SystemFew Special FilesUse CasesKernel Module Vs Kernel Built-inKernel Module Vs User ApplicationHow “insmod” worksHow “rmmod” works

Outline

Introduction

Let us start our discussion with a diagrammaticrepresentation of a Linux system

User Space Applications

Hardware

Operating System

Kernel

Kernel modules

What is Kernel

What is kernel?

- Central core of an Operating System

- Kernel is loaded first during booting and stays till the system is up. So image size should be minimum.

- Usually loaded into a protected memory area – Kernel Space.

- Kernel and BIOS are completely separate entity.

What is Kernel (Contd.)

Main Roles of kernel?

- Kernel executes jobs or handles interrupts etc. in Kernel Space.

- Kernel provides a set of portable, architecture and hardware independent Kernel APIs to allow user space applications to use the hardware resources.

- User Space applications request for basic services (Memory management; Process management; I/O management etc.) through system calls – Services provided by kernel.

- Kernel handles concurrent access and usage of hardware resources.

Kernel Space and User Space

Kernel codes [including kernel modules] execute in a separate address space [protected from being overwritten by external programs] with super-user privilege : This environment [Address space + privilege] is called Kernel Space

User applications run in another separate address space with lowest privilege mode : This environment is called User Space

Linux system can switch from User Space to Kernel Space, whenever an application issues a system call, or an hardware interrupt suspends the application process

Kernel code executing a system call executes in process context; whereas the code handling interrupts [Interrupt Handlers or Interrupt Service Routines] works asynchronously in Interrupt context.

Kernel CategoriesMicro kernel:

- Provides minimal services, such as defining Memory Address Space, IPC and CPU management- All other services such as Hardware management etc. are implemented separately as User Space Processes.- Examples: AIX, Mac OS X, MINIX etc.

Monolithic kernel:- Contains all the core functions of OS and device drivers- Some can load modules dynamically to extend kernel features on demand- Examples: Linux, FreeBSD etc.

Hybrid kernel:- Similar to micro kernel, except that they include additional code in kernel space so that such code can run more swiftly compared to if those code were made to run from User Space.- Examples: Windows etc.

Exo kernel:- Still experimental

Linux Kernel Sourcearch/<ARCH>:

- Architecture specific codeblock/:

- Block layer core COPYING:

- Linux copying conditions (GNU GPL).CREDITS:

- Linux main contributorscrypto:

- Cryptographic librariesDocumentation/:

- Kernel documentation. Don't miss it!drivers/:

- All device drivers except sound ones (usb, pci...)firmware/:

- Legacy: firmware images extracted from old drivers

Linux Kernel Source (Contd)fs/:

- Filesystems (fs/ext3/, etc.)include/:

- Kernel headersinclude/linux/:

- Linux kernel core headersinclude/uapi/:

- User space API headersinit/:

- Linux initialization (including main.c)ipc/:

- Code used for process communicationKbuild:

- Part of the kernel build system

Linux Kernel Source (Contd)Kconfig:

- Top level description file for conguration parameterskernel/:

- Linux kernel core (very small!)lib/:

- Misc library routines (zlib, crc32...)MAINTAINERS:

- Maintainers of each kernel part. Very useful!Makefile:

- Top Linux Makefile (sets arch and version)mm/:

- I Memory management code (small too!)net/:

- Network support code (not drivers)README:

- Overview and building instructions

Linux Kernel Source (Contd)REPORTING­BUGS:

- Bug report instructionssamples/:

- Sample code (markers, kprobes, kobjects...)scripts/:

- Scripts for internal or external usesecurity/:

- Security model implementations (SELinux...)sound/:

- Sound support code and driverstools/:

- Code for various user space tools (mostly C)usr/:

- Code to generate an initramfs cpio archivevirt/:

- Virtualization support (KVM)

Build your own Kernel[root@asl­host169]# make defconfig/** Creates a default .config file for kernel configuration, based on underlying

system*/[root@asl­host169]# make menuconfig/** Opens a text based GUI, where we can custom-configure the kernel as

per our requirement. The previous .config file is kept as backup with name .config.old, and a new .config file is created along with the modifications. Put a [*] against “Enable loadable module support”

*/[root@asl­host169]# make bzImage modules modules_install 

install/** Builds the kernel and its modules; Prepares various files like vmlinux, vmlinux.bin, vmlinuz, zImage, bzImage, System.map, initrd.img etc.

*/

Kernel Files

vmlinux:Binary image of Linux kernel in a Statically Linked Executable File Format.

vmlinux.bin: Same as vmlinux, but in a bootable raw binary file format. All symbols and relocation information is discarded.Binary image of Linux kernel in a Statically Linked Executable File Format. Generated from vmlinux by objcopy ­O binary vmlinux vmlinux.bin.

vmlinuz:Binary compressed [with zlib/LZMA/bzip2] vmlinux file. During bootup, it is decompressed to get the boot image.

Kernel Files (Contd.)

zImage:Prepared with make zImage. This is an old format for small kernels.

bzImage:Prepared with make bzImage. This big zImage was created while the kernel grew bigger enough, and accordingly the size of the binary image.

System.map:Static kernel symbol table.

initrd.img:A small file that does some initiations, and extracts and executes the actual kernel file.

Kernel Modules- Linux Kernel Module (LKM) are nothing but a piece of kernel code, that can be loaded as and when required to the running kernel. However, kernel must should “Enable loadable module support” [CONFIG_MODULES=y] for this.

- The prototype for the init and exit function of a module:init function: static int __init <init_fn_name>(void)exit function: static void __exit <exit_fn_name>(void)

- The __init and __exit Macros:__init macro: causes the init function to be discarded and its memoryfreed once the init function finishes for built-in codes, but not loadable modules.

__exit macro: causes the omission of the function when the module is built into the kernel, and has no effect for loadable modules.

These macros are defined in linux/init.h

Kernel Modules (Contd.)

Fine, it’s time for an example:

Step1: Write a HelloWorld module#include <linux/init.h>#include <linux/module.h>MODULE_LICENSE("Dual BSD/GPL");static int hello_init(void){        printk(KERN_ALERT "Hello, world\n");        return 0;}static void hello_exit(void){        printk(KERN_ALERT "Goodbye, cruel world\n");}module_init(hello_init);module_exit(hello_exit);

Kernel Modules (Contd.)Step2: Prepare a Makefile to build it to HelloWorld.ko kernel objectKERNELDIR=/lib/modules/$(shell uname ­r)/buildobj­m += HelloWorld.o

all:      make ­C $(KERNELDIR) M=$(PWD) modulesclean:      make ­C $(KERNELDIR) M=$(PWD) clean

Step3: Keep the HelloWorld.c and the Makefile in same directory, and issue make

[root@asl­host169 my_module]# makemake[1]: Entering directory `/home/kernel/linux­3.19'  CC [M]  /home/dibyajyoti/Study_mats/my_module/HelloWorld.o  Building modules, stage 2.  MODPOST 1 modules  CC      

/home/dibyajyoti/Study_mats/my_module/HelloWorld.mod.o  LD [M]  /home/dibyajyoti/Study_mats/my_module/HelloWorld.komake[1]: Leaving directory `/home/kernel/linux­3.19'

Kernel Modules (Contd.)

Step4: Load the module[root@asl­host169 my_module]# insmod HelloWorld.ko[root@asl­host169 my_module]# dmesg ­c[706754.319134] Hello, world[root@asl­host169 my_module]#

Step5: Unload the module[root@asl­host169 my_module]# rmmod HelloWorld[root@asl­host169 my_module]# dmesg ­c[706891.948856] Goodbye, cruel world[root@asl­host169 my_module]#

Step6: Bingo! The journey of our module is end :-)

Kernel Module Build SystemRef: <Kernel_Home>/Documentation/kbuild/modules.txt

- "kbuild" is the build system used by Linux kernel (Our example uses it)- Modules' build system should comply with this for easy compatibility to

changes in build infrastructure and pick right flags to gcc- Functionality for both in-tree and out-of-tree module building are

provided- External modules are supplied with Makefiles to hide complexity- The command to build an external module is:

$ make -C <path_to_kernel_src> M=$PWD- To build against the running kernel use:

$ make -C /lib/modules/`uname -r`/build M=$PWD- Then to install the module(s) just built, add the target

"modules_install" to the command:$ make -C /lib/modules/`uname -r`/build M=$PWD modules_install

Kernel Module Build System (Contd.)

- make -C $KDIR M=$PWD ($KDIR refers to the path of the kernel source directory.)

- “-C $KDIR”The directory where the kernel source is located. "make" will actually change to the specified directory when executing and will change back when finished.

- “M=$PWD”Informs kbuild that an external module is being built. The value given to "M" is the absolute path of the directory where the external module (kbuild file) is located.

Kernel Module Build System (Contd.)

- When building an external module, make line looks like (generally):make -C $KDIR M=$PWD [target]

“[target]” is optional, and can have following values:- “modules”

Default target for external modules. It has the same functionality as if no target was specified.

- “modules_install”Installs the external module(s). Default location is /lib/modules/<kernel_release>/extra/, but a prefix may be added with INSTALL_MOD_PATH

- “clean”Removes all generated files in the module directory only.

- “help”Lists the available targets for external modules.

Kernel Module Build System (Contd.)

- “obj-<y/m> := <module_name>.o”

kbuild system will build <module_name>.o from <module_name>.c,and, after linking, will result in the kernel module <module_name>.ko. The above line can be put in either a "Kbuild" file or a "Makefile.

NOTE: Many a times, we may encounter lines like:“obj-$(CONFIG_BT) += bluetooth/”Means, value of the tri-state “CONFIG_BT” ['m' if module; 'y' if built-in; 'n/<NULL>' if not defined] is appended to make the line obj-m / obj-y / not-defined correspondingly.

Kernel Module Build System (Contd.)

- When the module is built from multiple sources, an additional line isneeded listing the files:

“<module_name>-objs := <src1>.o <src2>.o ...”

Few Special Files

Once the module is built, many files are generated, like:

- <module_name>.ko : Actual kernel module file to “insmod”. “ko” Stands for Kernel Object.

- Module.symvers : When your module “exports” symbols using EXPORT_SYMBOL or its variants, those symbols are listed here along with the EXPORTer module name

- modules.order : This file tells the modules should be loaded in which order [Top-down ordering]

Use case

I built kernel 3.3 with mmc_core as module and kernel 3.9 with mmc_core as built-in.

Result:The Module.symvers of both contain the symbols.However, System.map of 3.9.0 contains symbols from mmc_core sub-section, while that of 3.3.0 does not contain the symbols.

- The file /proc/kallsyms in proc filesystem is the dynamic list of all the kernel symbols

Use case

Most of us probably faced:insmod: ERROR: could not insert module <module_name>.ko: Unknown

symbol in module- The above occurs for modules, but never originates for kernel built-in. Reason:- Kernel built-in are built during kernel compilation phase, and symbols

are resolved when preparing the kernel binary image. At this phase, all the symbol references are resolved

- Kernel modules are external agent, that refers to kernel symbols. When we call insmod utility to load the module, first the symbols are resolved. During this phase, if the kernel symbol is not available in /proc/kallsyms file, we face the above error (irritation!).

Kernel Modules Vs. Kernel built-inSimilarity:

- Both seats and executes in kernel space completely

- Both has direct access to whole kernel

- Once loaded, both have equal rights and responsibilities

Dissimilarity:

- Unlike module, kernel built-in is part of the kernel binary image

- kernel built-in can access the whole source [if not static function]. But modules can access kernel code only if they are any of macro / inline fn or symbol exported by kernel using EXPORT_SYMBOL or its variants

- Kernel built-in has lesser memory footprint compared to module

- Kernel built-in usually outperforms kernel modules

Kernel Modules Vs. User ApplicationsFull of Dissimilarities:

- Unlike Applications, Kernel modules do not define a main() function

- Kernel modules link only to kernel; Aplications link to libraries

- Kernel modules use different set of header files than user applications

- Kernel modules should avoid global variables, as those must be unique kernel-wide.

- Kernel module may be hardware specific; applications are generally not

- Kernel modules can be dynamically loaded.

- Kernel module runs in kernel space; application runs in user space

- Kernel modules typically are not sequential in nature.

- Kernel modules typically can be interrupted.

- Faults may be fatal to system for kernel codes.

How “insmod” worksSteps in brief:

- Allocates a buffer, and scales it up if required

- Opens the <module_name>.ko file [O_RDONLY mode], and copies [read] the entire file to the userspace buffer

- Now it issues init_module system call; C Compiler supported ABI assigns a unique number [128 for init_module] to it - eventually it maps to kernel call sys_init_module

Ref: File: arch/x86/include/generated/uapi/asm/unistd_32.h#define __NR_init_module 128

File: arch/x86/include/generated/asm/syscalls_32.h__SYSCALL_I386(128, sys_init_module, sys_init_module)

- However, sys_init_module code does not directly find any. It is defined by using the following macro [defined in kernel/module.c]

SYSCALL_DEFINE3(init_module, void __user *, umod, unsigned long, len, const char __user *, uargs);

How “insmod” works (Contd.)

- This call does the following major steps to actually load the module- copy_module_from_user() : Allocates memory in kernel space [vmalloc()]; & copies the userspace buffer to kernel space [copy_from_user()]

- load_module() : Does the following 1. various sanity checks [elf header etc.] 2. some basic setup [refcnt; MODINFO_ATTR field etc] (setup_modinfo()) 3. some preparation for relocation 4. copies the arguments 5. set module state as MODULE_STATE_COMING (complete_formation()) 6. setup of link to sysfs and 7. call do_init_module() : This actually calls the module initialization

function (<fn>) that is specified by our module using module_init(<fn>). Additionally, the module state is set to MODULE_STATE_LIVE

How “rmmod” works

Steps in brief:

- It determines the module name and checks if it is in use presently

- Now it issues delete_module system call; C Compiler supported ABI assigns a unique number [129 for delete_module] to it - eventually it maps to kernel call sys_delete_module

Ref: File: arch/x86/include/generated/uapi/asm/unistd_32.h#define __NR_delete_module 129

File: arch/x86/include/generated/asm/syscalls_32.h__SYSCALL_I386(129, sys_delete_module, sys_delete_module)

- However, sys_delete_module code does not directly find any. It is defined by using the following macro [defined in kernel/module.c]

SYSCALL_DEFINE2(delete_module, const char __user *, name_user, unsigned int, flags);

How “rmmod” works (Contd.)- This call does the following major steps to actually load the module

-Copies the module name from userspace - Traverse the list of modules and find the specified module by name - Checks if other modules are dependent on it - Synchronizes all asynchronous function calls. This function waits until all asynchronous functiona calls are done - free_module(): Frees up a module, its memory, removes from list etc. 1. unlinks the sys file system from the module 2. set module state tp MODULE_STATE_UNFORMED 3. Removes dynamic debug info. 4. Arch specific cleanup 5. Clear the unload stuff from module 6. Frees up any allocated parameters 7. Some more free up are done; Frees up the core containing the module structure 8. Frees up module memory – the kernel space buffer allocated during load