# Linux Kernel Deadlock Deadlocks are one of the most common synchronization problems encountered in operating systems and concurrent programming. Although the concept is frequently introduced in textbooks, observing it inside the Linux kernel provides a much deeper understanding of why proper lock ordering is essential. In this project, I built a **Linux Kernel Loadable Module (LKM)** that intentionally creates a deadlock using **two kernel threads** and **two mutexes**. The same module also contains a corrected implementation that eliminates the deadlock by enforcing a consistent lock acquisition order. The objective of this project is educational. It demonstrates not only how deadlocks occur, but also how to debug synchronization issues using kernel logs and how to manage kernel thread lifecycles correctly. --- # Project Overview The module supports two execution modes. | Mode | Description | |------|-------------| | `mode=1` | Intentionally creates a deadlock | | `mode=2` | Demonstrates the corrected implementation | Both modes use the same synchronization primitives but differ in the order in which mutexes are acquired. --- # Technologies Used - Linux Kernel Modules (LKM) - Kernel Threads (`kthread`) - Mutex Synchronization - Linux Kernel Logging (`pr_info`) - Module Parameters - `dmesg` - Kernel Build System --- # Understanding the Deadlock The demonstration creates two worker threads. The first thread acquires: ```text LockA ↓ LockB ``` The second thread acquires: ```text LockB ↓ LockA ``` Once each thread acquires its first mutex, both wait forever for the second mutex. Neither thread can proceed. This is the classic circular waiting condition that produces a deadlock. --- # Deadlock Demonstration The worker thread acquires the first mutex, waits briefly, and then attempts to acquire the second mutex. Example: ```c mutex_lock(&lockA); msleep(2000); mutex_lock(&lockB); ``` The second worker performs the opposite operation. ```c mutex_lock(&lockB); msleep(2000); mutex_lock(&lockA); ``` Running both threads simultaneously causes each thread to wait indefinitely. --- # Deadlock Prevention The solution is surprisingly simple. Both threads acquire the mutexes in exactly the same order. ```c mutex_lock(&lockA); mutex_lock(&lockB); ``` Since every thread follows the same locking sequence, circular waiting is eliminated. --- # Kernel Threads The module creates two kernel threads using `kthread_run()`. Example: ```c thread1 = kthread_run(worker1, NULL, "thread1"); ``` Each worker performs its task once and then exits. This project is intentionally kept small so that the synchronization behavior remains easy to observe. --- # Module Parameters The module behavior is controlled through a module parameter. ```bash sudo insmod deadlock_demo.ko mode=1 ``` or ```bash sudo insmod deadlock_demo.ko mode=2 ``` This makes it possible to compare the incorrect and corrected implementations without modifying the source code. --- # Building the Module Compile the module using the kernel build system. ```bash make ``` If Secure Boot is enabled, sign the module before loading it. ```bash sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file \ sha256 \ ~/kernel_keys/MOK.key \ ~/kernel_keys/MOK.crt \ deadlock_demo.ko ``` --- # Running the Demonstration Load the deadlock version. ```bash sudo insmod deadlock_demo.ko mode=1 ``` Monitor kernel messages. ```bash dmesg -w ``` Load the corrected implementation. ```bash sudo insmod deadlock_demo.ko mode=2 ``` Unload the module. ```bash sudo rmmod deadlock_demo ``` --- # Debugging with dmesg The Linux kernel provides detailed diagnostic information through `dmesg`. Useful commands include: ```bash dmesg | tail -100 ``` ```bash dmesg | grep deadlock_demo ``` Kernel logs make it much easier to understand thread execution, lock acquisition, and module initialization. --- # Two Important Bugs Encountered While implementing this demonstration, I encountered two issues that were **not** caused by the deadlock itself. ## 1. Incorrect use of `kthread_stop()` The worker threads in the corrected implementation naturally terminate after completing their work. Attempting to stop already-finished threads during module removal resulted in kernel warnings and crashes. The fix was to avoid calling `kthread_stop()` for one-shot worker threads. --- ## 2. Missing `IS_ERR()` Validation Initially, the return value of `kthread_run()` was not validated. Proper kernel programming requires checking whether thread creation failed. Example: ```c if (IS_ERR(thread1)) ``` Adding this validation makes the module more robust and avoids invalid pointer usage. A detailed analysis of both debugging sessions will be presented in a future article. --- # Key Concepts Covered - Linux Kernel Modules - Kernel Threads - Mutexes - Mutual Exclusion - Critical Sections - Deadlocks - Deadlock Prevention - Lock Ordering - Module Parameters - Kernel Logging - Thread Lifecycle - Basic Kernel Debugging --- # What I have Learned By studying this project,I have gained hands-on experience with: - Writing Linux Kernel Modules - Creating kernel threads - Synchronizing shared resources - Understanding why deadlocks occur - Preventing deadlocks using proper lock ordering - Reading kernel logs - Debugging synchronization problems --- # Future Articles This project is the first step in a larger Linux kernel synchronization series. Upcoming articles will provide a deeper discussion of: - Complete source code walkthrough - Thread lifecycle management - `kthread_run()` internals - `kthread_stop()` best practices - `IS_ERR()`, `ERR_PTR()`, and `PTR_ERR()` - Linux kernel debugging techniques - Kernel call trace analysis - Synchronization best practices --- # Conclusion Deadlocks are easier to understand when they can be reproduced in a controlled environment. This Linux Kernel Module demonstrates both an incorrect synchronization strategy and its corrected implementation using only two kernel threads and two mutexes. Although the project is intentionally small, it illustrates several important concepts that frequently appear in operating system design and Linux kernel development. Understanding synchronization primitives at this level provides a solid foundation for exploring more advanced topics such as semaphores, read-write locks, completions, wait queues, workqueues, RCU, and lock-free programming. --- ## GitHub Repository Source code for this project is available at: **https://github.com/aj333git/linux_kernel_deadlock_demo**