Introduction

When developing Linux kernel code, counters and object lifetime management can look deceptively similar.

Both may involve operations such as:

increment();
decrement();

But they solve fundamentally different problems.

Two important Linux kernel primitives are:

• atomic_t

• refcount_t

The distinction is simple:

atomic_t is primarily for atomic integer operations.

refcount_t is specifically for reference counting and object lifetime management.

Understanding this distinction becomes increasingly important when writing kernel modules that operate in concurrent environments.

1. The Core Idea

A kernel object can contain both an atomic counter and a reference counter:

struct demo_object {
    atomic_t  counter;
    refcount_t refs;
};

Although both fields contain numeric values, their meaning is different.

This semantic difference is the foundation of the entire demonstration.

2. What Is atomic_t?

atomic_t provides atomic operations on an integer value.

Common kernel APIs include:

For example:

atomic_inc(&obj->counter);

This performs an atomic increment of the counter.

The important word is:

atomic

The operation is designed for concurrent execution contexts so that the particular atomic update is performed as an atomic operation.

However, this does not mean that every operation involving the object automatically becomes thread-safe.

3. Typical Uses of atomic_t

atomic_t is useful when the value represents activity or statistics.

Examples include:

For example:

atomic_inc(&obj->counter);

could mean:

"Another request has been processed."

The counter represents activity, not ownership.

4. What Is refcount_t?

refcount_t is intended for reference counting.

Instead of asking:

How many operations happened?

it asks:

How many references currently keep this object alive?

Common APIs include:

The key concept is object lifetime.

5. Reference Counting Lifecycle

Consider an object that starts with one reference.

Object created
      |
      v
refs = 1
      |
      | another user obtains reference
      v
refs = 2
      |
      | one user releases reference
      v
refs = 1
      |
      | final reference released
      v
refs = 0
      |
      v
Object can be freed

The critical transition is:

1 -> 0

When the final reference is released, the object may become eligible for cleanup.

6. atomic_t vs refcount_t

These primitives should not be treated as interchangeable.

A useful mental model is:

The most important distinction is:

                 What does the number mean?
                           |
             +-------------+-------------+
             |                           |
             v                           v
       Activity/Event              Object Ownership
             |                           |
             v                           v
         atomic_t                    refcount_t

7. Demonstration 1: Basic atomic_t

The first demonstration treats atomic_t as a simple operation counter.

The counter starts at zero:

0

Five increments produce:

0 -> 1 -> 2 -> 3 -> 4 -> 5

An additional atomic addition of 10 produces:

5 -> 15

A decrement then produces:

15 -> 14

The important concept is not the final number.

The important concept is what the number represents.

Here it represents:

Operations / Activity

It does not determine whether an underlying object is still alive.

8. Demonstration 2: Basic refcount_t

Now consider a reference counter.

The reference count starts with one reference:

refs = 1

That initial reference represents ownership of the object.

When another user needs the object:

1 -> 2

When that user finishes:

2 -> 1

The object remains alive because one reference still exists.

Eventually, when the final reference is released:

1 -> 0

the object can reach its cleanup state.

The important idea is:

refcount_t
    |
    v
Object lifetime
    |
    v
Can the object still be safely used?

9. Demonstration 3: Using Both Together

A kernel object can contain both an operation counter and a reference count.

For example:

struct demo_object {
    atomic_t   counter;
    refcount_t refs;
};

Their meanings are independent:

Suppose the object is in this state:

There is no requirement for these values to match.

For example:

Operations = 5
References = 2

is perfectly meaningful.

Five operations do not imply five references.

Similarly:

References = 2

does not imply that exactly two operations have occurred.

They represent different aspects of the object's state.

10. Demonstration 4: Requests and Object Lifetime

Consider a kernel object that processes requests.

Every request can increment an operation counter:

atomic_inc(&object->counter);

The counter might evolve as follows:

At the same time, another user may acquire a reference to the object.

This produces two independent forms of tracking:

Conceptually:

                  Kernel Object
                       |
             +---------+---------+
             |                   |
             v                   v
      Operation activity     Object lifetime
             |                   |
             v                   v
         atomic_t            refcount_t
             |                   |
             v                   v
       "What happened?"     "Who still needs it?"

This separation of responsibilities is important in concurrent kernel programming.

11. Demonstration 5: Multiple References

Suppose an object starts with:

refs = 1

Three additional references are acquired:

1 -> 2 -> 3 -> 4

The reference count is now:

refs = 4

Each additional reference represents another entity that currently needs the object to remain alive.

Those references must eventually be released.

After releasing the three temporary references:

4 -> 3 -> 2 -> 1

The original reference remains.

Finally, when that last reference is released:

1 -> 0

the object can be cleaned up.

12. The Reference Counting Rule

One of the most important rules when working with reference counting is:

Every acquired reference must eventually be released.

Conceptually:

The basic lifecycle is:

Acquire
   |
   v
refs + 1
   |
   v
Use object
   |
   v
Release
   |
   v
refs - 1
   |
   +-------------------+
   |                   |
   | refs > 0          | refs == 0
   |                   |
   v                   v
Object remains       Cleanup
alive

An unbalanced reference can prevent an object from reaching its cleanup state.

This is one reason reference-counting bugs can be difficult to diagnose.

13. Object Lifetime

The demonstration dynamically allocates its object.

The lifecycle can be understood as:

The overall lifecycle looks like this:

+----------------------+
| Object allocated     |
+----------+-----------+
           |
           v
+----------------------+
| Initialize counters  |
| and references       |
+----------+-----------+
           |
           v
+----------------------+
| Object is usable     |
+----------+-----------+
           |
           v
+----------------------+
| References exist     |
+----------+-----------+
           |
           v
+----------------------+
| References released  |
+----------+-----------+
           |
           v
+----------------------+
| refcount reaches 0   |
+----------+-----------+
           |
           v
+----------------------+
| Object can be freed  |
+----------------------+

The critical relationship is:

Object allocated
       |
       v
References exist
       |
       v
Object is usable
       |
       v
Final reference released
       |
       v
Reference count reaches zero
       |
       v
Object can be freed

14. Why Not Use atomic_t for Everything?

At first glance, it may seem reasonable to use atomic_t for both operation counters and object references.

For example:

atomic_t counter;

But compare it with:

refcount_t refs;

The second declaration communicates much more information.

atomic_t counter;
        |
        +--> This is an atomic integer counter.


refcount_t refs;
        |
        +--> This represents references to an object.

Reference counting has semantic requirements related to ownership and lifetime.

Using the appropriate kernel primitive therefore makes the code:

• easier to understand

• clearer to maintain

• more explicit about ownership

• better aligned with the intended kernel abstraction

The type itself communicates intent.

15. Atomic Does Not Mean the Entire Program Is Thread-Safe

This is an important concept.

An atomic operation does not automatically make an entire algorithm thread-safe.

Suppose a structure contains several fields:

struct demo_object {
    atomic_t counter;
    int state;
    void *data;
};

Making this operation atomic:

atomic_inc(&obj->counter);

does not automatically protect:

obj->state
obj->data

from concurrent access.

Atomicity applies to the particular atomic operation.

A larger algorithm may still require synchronization.

16. Kernel Synchronization Primitives

Linux provides multiple synchronization mechanisms, each intended for different problems.

Conceptually:

                 Concurrency Problem
                         |
          +--------------+--------------+
          |              |              |
          v              v              v
      Counter        Critical       Read-heavy
      update         section        access
          |              |              |
          v              v              v
      atomic_t       Spinlock          RCU
                         |
                         v
                       Mutex

The correct primitive depends on what is being protected and what the execution context allows.

17. Kernel Headers

The demonstration uses several kernel headers.

For example:

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/atomic.h>
#include <linux/refcount.h>
#include <linux/slab.h>

These headers provide the infrastructure required by the demonstration module.

18. Memory Management

The example dynamically allocates its demonstration object.

The relevant memory-management APIs are:

It is important to distinguish memory allocation from reference counting.

Therefore:

kmalloc()
   |
   v
Memory exists
   |
   v
refcount_t
   |
   v
Track who still references it
   |
   v
refs == 0
   |
   v
kfree()

A reference count does not allocate memory.

Likewise, kmalloc() does not track object ownership.

The code must connect these mechanisms correctly.

19. Module Lifecycle

A Linux kernel module normally has an initialization path and an exit path.

The demonstration performs its examples during initialization.

The dynamically allocated object is eventually released during module cleanup after the reference count reaches its final state.

Conceptually:

module_init()
     |
     v
Allocate object
     |
     v
Initialize atomic_t
     |
     v
Initialize refcount_t
     |
     v
Run demonstrations
     |
     v
module_exit()
     |
     v
Release object

20. A Practical Mental Model

A simple way to remember the difference is to ask what the number means.

                    What does the number represent?
                                  |
                    +-------------+-------------+
                    |                           |
                    v                           v
             Activity / Events             Ownership / Lifetime
                    |                           |
                    v                           v
                atomic_t                    refcount_t
                    |                           |
                    v                           v
             "How many?"                 "Who still needs it?"

Examples:

How many packets were processed?
        |
        +--> atomic_t


How many requests occurred?
        |
        +--> atomic_t


How many references exist?
        |
        +--> refcount_t


Can the object be freed?
        |
        +--> refcount_t

21. Common Conceptual Mistakes

Mistake 1: Treating atomic_t as a lifetime mechanism

An atomic counter can count events, but that does not automatically mean it correctly models object ownership.

atomic_t counter;

does not communicate the same lifetime semantics as:

refcount_t refs;

Mistake 2: Assuming atomic means thread-safe

This:

atomic_inc(&obj->counter);

only makes that atomic operation atomic.

It does not automatically protect:

obj->state;
obj->data;
obj->other_field;

Mistake 3: Assuming counters should match

Consider:

Operations = 100
References = 2

There is nothing inherently wrong with this.

The values represent different concepts.

100 operations
      !=
2 references

Mistake 4: Forgetting to release references

If a reference is acquired:

refs + 1

it must eventually be released:

refs - 1

Otherwise the object may never reach:

refs == 0

and therefore may never reach its intended cleanup state.

22. atomic_t and refcount_t in One Object

A useful conceptual example is:

struct demo_object {
    atomic_t   operations;
    refcount_t refs;
};

The object now has two independent dimensions of state:

                    demo_object
                         |
              +----------+----------+
              |                     |
              v                     v
        operations                refs
              |                     |
              v                     v
          atomic_t              refcount_t
              |                     |
              v                     v
       Activity tracking      Lifetime tracking

For example:

operations = 500
refs       = 3

This means:

500 operations have been counted
3 references currently exist

It does not mean:

500 references

or:

3 operations

The two counters have independent meanings.

23. Key Lessons

The most important concepts from this project are:

The central distinction is:

atomic_t
   |
   +--> Atomic counting
   |
   +--> Operations / events / statistics


refcount_t
   |
   +--> Reference counting
   |
   +--> Object ownership / lifetime

24. Where to Go Next

This demonstration is intentionally small.

Once the basic concepts are clear, the next step is to move from isolated API demonstrations toward realistic Linux kernel concurrency and lifetime-management patterns.

A useful progression is:

The progression can be visualized as:

atomic_t
   |
   v
refcount_t
   |
   v
Object Lifetime
   |
   v
Kernel Threads
   |
   v
Spinlocks
   |
   v
Mutexes
   |
   v
Character Devices
   |
   v
Concurrent User-Space Access
   |
   v
kref
   |
   v
RCU
   |
   v
Memory Ordering
   |
   v
Real Kernel Subsystem Patterns

This moves the learning process from simple primitives toward the concurrency and lifetime-management techniques used in larger kernel components.

25. Final Takeaway

The entire distinction can be summarized with two questions:

Or, even more simply:

              atomic_t
                  |
                  v
       "How many operations?"
                  |
                  v
          Atomic counting


              refcount_t
                  |
                  v
       "Who still references it?"
                  |
                  v
          Object lifetime

A well-designed kernel component should use the primitive that matches the meaning of the data.

atomic_t
    =
atomic counting


refcount_t
    =
reference counting
    +
object lifetime

Understanding this distinction provides a strong foundation for more advanced Linux kernel topics such as:

• concurrency

• locking

• reference ownership

• character drivers

• kref

• RCU

• memory ordering

• lifetime management

The key lesson is therefore not simply how to call the APIs.

It is understanding why the kernel provides different primitives for different meanings.

GitHub Repository

The complete source code, build files, and module implementation are available here:

linux_kernel_atomic_refcount_2

Explore the repository to see the concepts demonstrated in an actual Linux kernel module.

Concurrent execution is an important concept in operating systems and kernel development. Modern processors can execute multiple tasks concurrently, often across multiple CPU cores. When multiple execution contexts access the same shared resource, synchronization becomes necessary to prevent inconsistent or unexpected results.

Linux provides several synchronization mechanisms for kernel development. One of the commonly used mechanisms is the spinlock.

This article explains the concept of combining Linux kernel threads with a spinlock-protected shared counter. The focus is on understanding the design, synchronization mechanism, execution flow, and expected behavior rather than implementation details.

What Is a Linux Kernel Thread?

A Linux kernel thread is a thread that executes entirely within kernel space.

Kernel threads are useful for performing background work that is managed by the kernel rather than directly by a user-space application.

A kernel thread:

• Runs in kernel space.
• Is scheduled by the Linux scheduler.
• Can execute concurrently with other kernel threads.
• Can access kernel data structures and resources.
• Can be started and stopped by kernel code.
• Can execute independently of a normal user-space process.

Kernel threads are commonly used for background kernel activities such as:

• Deferred processing.
• Device-related operations.
• Background maintenance.
• Resource management.
• Kernel subsystems that require independent execution contexts.

Why Synchronization Is Necessary

Consider a shared counter accessed by two kernel threads.

Both threads perform repeated increments of the same variable.

At first glance, incrementing a counter appears to be a simple operation. However, an increment is conceptually composed of multiple steps:

1. Read the current value.
2. Add one to the value.
3. Write the new value back.

When two threads perform these operations concurrently, their actions can overlap.

For example:

Both threads may calculate their new value from the same original value.

As a result, one increment can effectively overwrite another increment.

This situation is known as a race condition.

What Is a Race Condition?

A race condition occurs when the final result of an operation depends on the timing or ordering of concurrent execution.

Race conditions are particularly dangerous in kernel programming because kernel code often works with shared resources used by multiple execution contexts.

A race condition can result in:

• Incorrect counter values.
• Corrupted data.
• Inconsistent state.
• Difficult-to-reproduce bugs.
• Unexpected system behavior.

The problem is not necessarily that either thread is incorrect individually. The problem is that both threads are accessing shared state without sufficient synchronization.

The Role of a Spinlock

A spinlock provides mutual exclusion.

Its purpose is to ensure that only one execution context can enter a protected critical section at a time.

The basic concept is:

1. A thread attempts to acquire the lock.
2. If the lock is available, the thread acquires it.
3. The thread accesses the protected resource.
4. The thread releases the lock.
5. Another waiting thread can acquire the lock.

This prevents two threads from modifying the same protected data simultaneously.

Critical Section

A critical section is a portion of code that accesses shared data and therefore must be protected from concurrent modification.

In the counter example, the critical section contains the counter update.

The conceptual execution becomes:

The important property is that the shared counter update is serialized.

How the Two Threads Work

The example uses two kernel threads.

Each thread performs a fixed number of counter updates.

The threads share:

• A common counter.
• A common spinlock.

Each thread also maintains its own individual count.

This provides two different types of information:

• The shared counter shows the total number of protected operations.
• The individual counters show how many operations each thread performed.

This makes the example useful for understanding both synchronization and concurrent execution.

Shared Data and Private Data

The design can be divided into shared and thread-specific data.

The shared counter requires synchronization because both threads modify it.

Why the Spinlock Protects the Counter

Without synchronization, both threads can access the counter at the same time.

With a spinlock, access becomes mutually exclusive.

The conceptual relationship is:

Two kernel threads → One shared counter → One spinlock → Controlled access

The lock does not prevent the threads from existing simultaneously.

Instead, it controls when they are allowed to access the protected resource.

This distinction is important.

Spinlocks and Waiting

A spinlock is different from a traditional sleeping synchronization mechanism.

When a thread cannot immediately acquire a spinlock, it waits by repeatedly checking the lock rather than putting itself to sleep.

This behavior is called spinning.

Spinning can be useful when:

• The protected critical section is very short.
• The expected waiting time is small.
• Sleeping is not appropriate in the current execution context.

However, spinning also consumes CPU resources while waiting.

Therefore, spinlocks should generally protect short critical sections.

Why the Critical Section Should Be Small

A spinlock should not be held for unnecessarily long periods.

Suppose Thread 1 acquires a spinlock and performs a lengthy operation while holding it.

Thread 2 cannot enter the protected section during that period and may continuously spin while waiting.

This wastes CPU time.

A good synchronization design therefore aims to:

• Acquire the lock as late as practical.
• Perform only necessary protected operations.
• Release the lock as soon as possible.
• Avoid lengthy operations while holding the lock.

In the counter example, the protected operation is intentionally small, making it suitable for demonstrating a spinlock.

Initial Counter Update

The combined example also performs a protected counter update during module initialization.

This demonstrates that spinlocks are not limited to kernel-thread synchronization.

A spinlock can protect shared kernel data whenever multiple execution contexts could potentially access that data.

The initial update establishes a starting value before the worker threads perform their operations.

Final Counter Calculation

The example performs one initial increment.

Each of the two worker threads then performs one hundred thousand increments.

Therefore, the expected total is:

The final counter therefore demonstrates that the protected updates were successfully preserved.

Individual Thread Counters

The example also maintains an individual count for each worker.

The expected values are:

These counters provide additional visibility into the work performed by each kernel thread.

They also help verify that both threads completed their expected workloads.

Kernel Thread Lifecycle

A kernel thread has a lifecycle similar to other kernel-managed execution contexts.

The general lifecycle is:

1. Module initialization begins.
2. The shared state is initialized.
3. The first worker thread is created.
4. The second worker thread is created.
5. Both threads execute concurrently.
6. Each thread performs its assigned work.
7. Each thread finishes.
8. The module can later be unloaded.
9. Cleanup ensures the thread resources are properly handled.

This lifecycle demonstrates an important principle of kernel programming:

Resources created by a kernel module must be managed carefully and cleaned up appropriately.

Stopping Kernel Threads

The worker design also supports a stop request.

A kernel thread should not simply be assumed to terminate immediately when the module is being removed.

Instead, the thread can periodically check whether it has been asked to stop.

This creates a cooperative termination mechanism.

The general concept is:

• The module requests that the worker stop.
• The worker notices the stop request.
• The worker exits its execution loop.
• The worker returns.
• Cleanup continues.

This is safer than assuming that a running kernel thread can simply be terminated from outside.

Error Handling

Kernel programming requires careful error handling because resource creation can fail.

For example, the first thread may be created successfully while the second thread fails to start.

In that situation, the module should not simply abandon the first thread.

The appropriate design is:

1. Detect the second thread creation failure.
2. Stop the already-created first thread.
3. Clean up the partially initialized state.
4. Return the appropriate error.

This is an example of failure-path cleanup.

Good kernel code should consider both:

• The successful execution path.
• The partial-failure path.

Why Error Handling Matters More in Kernel Code

Errors in kernel code can have consequences beyond a single application.

Poor cleanup can result in:

• Resource leaks.
• Threads continuing to run unexpectedly.
• Invalid references.
• Kernel instability.
• Difficult-to-debug failures.

Therefore, kernel modules should carefully manage every resource they create.

Spinlock Versus No Synchronization

The difference can be summarized conceptually:

The spinlock does not make the threads execute sequentially overall.

It only serializes access to the protected critical section.

Spinlock Versus Mutex

Spinlocks and mutexes are both synchronization mechanisms, but they have different characteristics.

The choice depends on the execution context and the expected duration of the critical section.

A spinlock is particularly useful when the protected operation is short and sleeping is not appropriate.

Important Kernel Programming Considerations

When using spinlocks in Linux kernel code, several principles are important.

Keep Critical Sections Short

Long critical sections increase contention and can waste CPU resources.

Do Not Sleep While Holding a Spinlock

A spinlock is designed for atomic, non-sleeping sections of kernel code.

Protect the Correct Data

The lock must protect every access that requires synchronization.

Protecting only some accesses can still leave race conditions.

Release the Lock

Every successful lock acquisition must have a corresponding release.

Handle Failure Paths

If initialization partially succeeds, previously allocated or started resources must be cleaned up.

What This Example Demonstrates

The small kernel module combines several important operating-system concepts:

• Kernel module lifecycle.
• Kernel threads.
• Shared memory.
• Race conditions.
• Mutual exclusion.
• Spinlocks.
• Critical sections.
• Thread synchronization.
• Thread termination.
• Error handling.
• Resource cleanup.

Because the example uses a simple counter, the synchronization behavior is easy to understand without introducing unnecessary kernel subsystems.

Conceptual Execution Flow

The overall flow can be summarized as:

Module initialization

The module initializes its shared state and performs an initial protected update.

Thread creation

Two worker threads are created.

Concurrent execution

Both threads begin performing their assigned operations.

Synchronization

Before modifying the shared counter, each thread must obtain the spinlock.

Critical section

The counter and corresponding thread-specific statistics are updated.

Lock release

The thread releases the spinlock so another thread can access the protected data.

Thread completion

Both workers eventually complete their workloads.

Module cleanup

The module ensures that the worker threads have stopped and then reports the final statistics.

Key Learning Points

The most important lessons from this example are:

• Concurrent execution creates the possibility of race conditions.
• Shared kernel data must be synchronized appropriately.
• A spinlock provides mutual exclusion for a critical section.
• Spinlocks are especially useful for short, non-sleeping critical sections.
• Kernel threads can execute concurrently and access shared kernel state.
• Thread termination should be handled cooperatively.
• Kernel modules must clean up resources during failure and removal.
• Synchronization protects data; it does not eliminate concurrency.
• A good critical section should be as small as practical.
• Correct synchronization makes concurrent behavior predictable.

Conclusion

Linux kernel threads provide a mechanism for executing kernel-level work concurrently. While concurrency improves flexibility and allows multiple activities to progress independently, it also introduces synchronization challenges.

A shared counter is a simple but effective example of this problem. Two kernel threads can access the same variable simultaneously, creating the possibility of a race condition. A spinlock solves this problem by ensuring that only one thread at a time can enter the critical section responsible for modifying the shared data.

The combined design demonstrates more than just how a spinlock works. It also illustrates kernel-thread lifecycle management, error handling, protected initialization and cleanup, and the importance of keeping synchronization regions small.

The broader lesson is fundamental to kernel development:

Concurrency requires synchronization whenever multiple execution contexts access shared mutable state.

Understanding this principle provides a strong foundation for studying more advanced Linux kernel synchronization mechanisms and concurrent operating-system design.

GitHub Repository: 👉 spin_lock_1 Explore the complete source code, build files, and module implementation on GitHub.

Priority inversion is a classic scheduling problem in real-time and concurrent systems.

It occurs when a high-priority task is forced to wait for a lower-priority task, while a medium-priority task can continue executing.

The basic model uses three tasks:

The important distinction is:

Priority inversion is the problem. Priority inheritance is one mechanism used to mitigate it.

This article explores the concept from a Linux kernel perspective using kernel threads and the Linux rt_mutex mechanism.

1. What Is Priority Inversion?

Suppose a low-priority task obtains a mutex.

A high-priority task subsequently needs the same mutex.

The high-priority task cannot continue because the resource is currently owned by LOW.

The situation becomes problematic when MEDIUM gets CPU time while LOW is unable to finish its critical section.

The logical sequence is:

1. LOW acquires the mutex.

2. HIGH attempts to acquire the mutex.

3. HIGH blocks.

4. MEDIUM performs unrelated work.

5. LOW is delayed.

6. HIGH remains blocked.

7. LOW eventually releases the mutex.

8. HIGH can finally continue.

The surprising part is that HIGH has a higher priority than MEDIUM and LOW, yet MEDIUM can indirectly contribute to HIGH's delay.

That is the essence of priority inversion.

2. Priority Inversion vs Priority Inheritance

These two terms should not be confused.

The important relationship is:

Priority inversion = problem
Priority inheritance = mitigation

3. The Classic Example

Consider:

HIGH    = 80
MEDIUM  = 50
LOW     = 10

LOW acquires a shared resource first.

HIGH later needs the same resource.

HIGH therefore blocks.

If MEDIUM can execute while LOW is waiting for CPU time, HIGH's effective waiting time can increase.

Conceptually:

LOW owns resource
        ↓
HIGH needs resource
        ↓
HIGH blocks
        ↓
MEDIUM executes
        ↓
LOW is delayed
        ↓
HIGH remains blocked

This is why priority inversion matters in systems where predictable response time is important.

4. What Is Priority Inheritance?

Priority inheritance addresses the dependency between the waiting task and the lock owner.

If HIGH is waiting for a mutex owned by LOW, LOW can temporarily inherit HIGH's priority.

Conceptually:

LOW priority: 10
        ↓
HIGH waits
        ↓
LOW temporarily inherits HIGH's priority
        ↓
LOW completes critical section
        ↓
LOW releases mutex
        ↓
HIGH continues
        ↓
LOW returns to its original priority

The goal is straightforward:

Allow the lower-priority lock owner to finish the critical section sooner so that the higher-priority waiter can proceed.

5. Why rt_mutex Matters in Linux

The Linux kernel provides rt_mutex as a real-time mutex implementation with priority-inheritance support.

A simplified example is:

static struct rt_mutex lock;

rt_mutex_init(&lock);

rt_mutex_lock(&lock);

/* critical section */

rt_mutex_unlock(&lock);

The important API calls are:

• rt_mutex_init()

• rt_mutex_lock()

• rt_mutex_unlock()

In the demonstration module, the shared lock is declared as:

static struct rt_mutex pi_lock;

and initialized during module loading.

6. Kernel Threads

The demonstration uses Linux kernel threads rather than ordinary user-space processes.

The three conceptual participants are:

• LOW

• HIGH

• MEDIUM

Kernel threads can be created using APIs such as:

kthread_run(low_thread, NULL, "pi_low");

The returned task_struct pointer allows the module to retain a reference to the created kernel thread.

For example:

static struct task_struct *low_task;

The same approach is used for HIGH and MEDIUM.

7. The LOW Thread

LOW is responsible for acquiring the rt_mutex.

The important operation is:

rt_mutex_lock(&pi_lock);

After acquiring the lock, LOW enters its critical section.

The demonstration deliberately keeps LOW holding the mutex for some time so that HIGH can attempt to acquire it.

The important relationship is therefore:

LOW owns the resource while HIGH needs it.

The module also prints scheduling information before and during the interaction.

8. The HIGH Thread

HIGH attempts to acquire the same mutex:

rt_mutex_lock(&pi_lock);

If LOW already owns the mutex, HIGH blocks.

The conceptual state is:

HIGH
  ↓
waiting for mutex
  ↓
LOW owns mutex

This is the critical point at which priority inheritance becomes relevant.

After LOW releases the mutex, HIGH can acquire it and continue.

9. The MEDIUM Thread

MEDIUM represents independent work.

It does not need the shared mutex.

Its purpose is to model the classic third participant in a priority-inversion scenario.

The simplified model is:

MEDIUM is therefore important because it illustrates how a task that has no direct relationship with the shared resource can still affect HIGH's waiting time.

10. A Linux 5.15 Constraint

There is an important implementation detail in this particular experiment.

The target environment is Ubuntu Linux 5.15.x.

An earlier version attempted to use:

sched_setscheduler_nocheck()

to assign arbitrary scheduler priorities.

However, that scheduler interface was not exported for use by the external loadable module in the target Ubuntu kernel.

As a result, the module cannot simply depend on that internal scheduler interface.

The implementation therefore uses exported helpers such as:

sched_set_fifo_low(low_task);
sched_set_fifo(high_task);

This is an important Linux kernel development lesson:

An API existing somewhere inside the kernel source does not automatically mean an out-of-tree module can call it.

11. Theoretical Priorities vs Actual Scheduler State

The textbook example commonly uses:

LOW    = 10
MEDIUM = 50
HIGH   = 80

However, those numbers should not be confused with the actual scheduler state of this module.

The current implementation uses the scheduler helpers available to the external module.

Therefore, the project deliberately distinguishes between:

Conceptual model

LOW    = 10
MEDIUM = 50
HIGH   = 80

Actual implementation

LOW    → sched_set_fifo_low()
HIGH   → sched_set_fifo()
MEDIUM  → normal kernel thread

The module prints actual scheduling fields such as:

• current->pid

• current->policy

• current->prio

• current->normal_prio

This is preferable to assuming that a theoretical priority value is the actual runtime value.

12. Standard Ubuntu Kernel vs PREEMPT_RT

Another important limitation is kernel configuration.

The target environment is a standard/non-PREEMPT_RT Ubuntu kernel.

That means this project should primarily be viewed as a Linux kernel rt_mutex and priority-inheritance learning experiment, rather than a deterministic real-time benchmark.

The exact timing and scheduling behavior can depend on:

• Kernel configuration

• Scheduler behavior

• CPU topology

• System load

• Preemption configuration

• Timing of msleep()

• Scheduling policy

For a strict textbook real-time demonstration with controlled priorities such as 10 / 50 / 80, a PREEMPT_RT kernel or an appropriately designed user-space POSIX experiment can be a better environment.

13. Why msleep() Is Used

The demonstration uses delays such as:

msleep(1000);

These delays are not intended to provide precise real-time scheduling.

Their purpose is to create enough temporal separation for the demonstration:

• LOW gets time to start.

• HIGH gets time to start.

• LOW obtains the mutex.

• HIGH subsequently attempts the mutex.

• MEDIUM begins its work.

Therefore, msleep() should be understood as a demonstration timing mechanism, not a real-time synchronization mechanism.

14. Module Lifecycle

A kernel module has an important lifecycle:

Build
  ↓
Load
  ↓
Initialize
  ↓
Create kernel threads
  ↓
Run demonstration
  ↓
Synchronize completion
  ↓
Stop threads
  ↓
Unload

The module entry point is:

static int __init pi_demo_init(void)

The module exit point is:

static void __exit pi_demo_exit(void)

These are connected using:

module_init(pi_demo_init);
module_exit(pi_demo_exit);

15. Why Module Cleanup Matters

Kernel thread lifecycle management is especially important.

An earlier version of the project experienced a kernel Oops during:

sudo rmmod pi_demo

The reported instruction pointer was associated with:

kthread_stop()

This highlighted an important kernel-programming rule:

Creating a kernel thread is only half of the problem. You must also correctly manage its entire lifecycle.

A module must carefully handle:

• Thread creation

• Thread execution

• Thread blocking

• Thread termination

• kthread_stop()

• Module removal

16. Using Completion for Safe Cleanup

The current design uses a kernel completion object:

static DECLARE_COMPLETION(demo_done);

HIGH signals completion after the actual demonstration:

complete(&demo_done);

The module cleanup path waits for that event:

wait_for_completion(&demo_done);

This gives the module a clear synchronization point before it proceeds with cleanup.

The lifecycle is therefore:

1. Start demonstration.

2. HIGH reaches completion.

3. HIGH signals demo_done.

4. Module cleanup waits for completion.

5. Cleanup stops the threads.

6. Module is unloaded.

This is safer than blindly stopping threads without considering their current state.

17. task_struct and Scheduler Information

The module stores thread references using:

struct task_struct *

The current executing task can be accessed through:

current

The demonstration prints scheduler-related information from the current task.

For example:

pr_info("PID=%d policy=%d prio=%d normal_prio=%d\n",
        current->pid,
        current->policy,
        current->prio,
        current->normal_prio);

This is useful because kernel experiments should distinguish between:

• what the source code intends,

• what the scheduler interface requests,

• and what the kernel actually reports.

18. Observing the Experiment with dmesg

Kernel modules normally use kernel logging APIs rather than printf().

The demonstration uses messages such as:

pr_info("PI-DEMO: HIGH trying to acquire rt_mutex\n");

You can filter the kernel log with:

sudo dmesg | grep PI-DEMO

For live output:

sudo dmesg -w

This allows the experiment to be observed while the kernel threads execute.

19. Useful Commands

Build

make

Check the module

ls -lh pi_demo.ko

Load

sudo insmod ./pi_demo.ko

Inspect output

sudo dmesg | grep PI-DEMO

Follow output live

sudo dmesg -w

Remove

sudo rmmod pi_demo

Check whether it is loaded

lsmod | grep pi_demo

Inspect recent messages

sudo dmesg | tail -30

Inspect the complete PI demonstration

sudo dmesg | grep -A80 -B20 "PI-DEMO"

Search for kernel problems

sudo dmesg | grep -E "BUG:|Oops:|WARNING:|PI-DEMO|kthread_stop" | tail -100

20. Recommended Laboratory Workflow

A clean experiment can follow this sequence:

Step 1 — Build

make

Step 2 — Verify the module

ls -lh pi_demo.ko

Step 3 — Load it

sudo insmod ./pi_demo.ko

Step 4 — Observe the output

sudo dmesg | grep PI-DEMO

or:

sudo dmesg -w

Step 5 — Remove it after completion

sudo rmmod pi_demo

Step 6 — Verify cleanup

sudo dmesg | tail -30

Step 7 — Verify module state

lsmod | grep pi_demo

21. What This Project Teaches

This relatively small module touches several important Linux kernel concepts.

Kernel modules

• module_init()

• module_exit()

• MODULE_LICENSE()

• MODULE_AUTHOR()

• MODULE_DESCRIPTION()

• MODULE_VERSION()

Kernel threads

• kthread_run()

• kthread_stop()

• kthread_should_stop()

• struct task_struct

Synchronization

• rt_mutex

• rt_mutex_init()

• rt_mutex_lock()

• rt_mutex_unlock()

Completion

• DECLARE_COMPLETION()

• complete()

• wait_for_completion()

Scheduling

• sched_set_fifo()

• sched_set_fifo_low()

• current->policy

• current->prio

• current->normal_prio

Debugging

• pr_info()

• pr_err()

• dmesg

• Kernel Oops

• Kernel warnings

• Thread cleanup

22. Common Conceptual Mistakes

Mistake 1: Priority inversion means LOW always runs before HIGH

Not exactly.

The problem is the resource dependency.

HIGH is blocked because LOW owns a resource that HIGH needs.

Mistake 2: Priority inheritance means LOW permanently becomes HIGH priority

No.

The inheritance is temporary and associated with the lock dependency.

After the relevant resource is released, the inherited priority can be removed.

Mistake 3: rt_mutex makes the entire system real-time

No.

Using rt_mutex does not automatically transform a standard Ubuntu kernel into a deterministic real-time operating system.

Kernel configuration and scheduling behavior still matter.

Mistake 4: sched_set_fifo() lets an external module select any priority number

No.

The exported helper does not provide arbitrary numeric priority assignment such as:

10
50
80

That was one of the important constraints encountered during development.

Mistake 5: If the code compiles, rmmod is automatically safe

No.

Module cleanup is part of kernel correctness.

Thread lifecycle bugs can surface only during module removal.

23. Priority Inheritance in One Sentence

Priority inheritance temporarily raises the effective priority of a lower-priority task holding a resource needed by a higher-priority task, helping the lock owner finish and release the resource sooner.

24. Final Takeaway

Priority inversion demonstrates why synchronization and scheduling cannot be studied completely independently.

A mutex determines who can access a resource.

The scheduler determines which runnable task gets CPU time.

Priority inheritance connects these two concerns when a high-priority task is blocked by a lower-priority lock owner.

In Linux, rt_mutex provides the kernel mechanism used for priority inheritance.

This project combines:

The most important lesson is not simply how to call rt_mutex_lock().

It is understanding the complete relationship between:

task → scheduler → resource → mutex → blocking → priority inheritance → critical section → unlock → task lifecycle → module cleanup.

GitHub Repository

Priority Inheritance Repository: 👉 priority_inheritance

Explore the complete source code, build files, and module implementation.

Priority inversion is a classic concurrency and real-time scheduling problem in operating systems.

The basic situation is simple:

• A LOW-priority thread owns a mutex.
• A HIGH-priority thread needs the same mutex.
• HIGH therefore has to wait for LOW.
• Meanwhile, a MEDIUM-priority thread can consume CPU time.
• LOW may be delayed from running and releasing the mutex.
• As a result, HIGH can remain blocked longer than expected.

The important idea is that the problem is not simply that HIGH is waiting for LOW.

The classic priority-inversion scenario is:

HIGH waits for LOW, while MEDIUM prevents LOW from running.

This creates an indirect dependency between HIGH and MEDIUM.

1. Start With the Mutex

A mutex provides mutual exclusion around a critical section.

In Linux kernel code, a basic mutex operation looks like:

mutex_lock(&my_mutex);

/* critical section */

mutex_unlock(&my_mutex);

Only one thread can own the mutex at a time.

If another thread calls mutex_lock() while the mutex is already owned, that thread has to wait.

For example:

LOW  -> owns mutex
HIGH -> needs mutex -> waits

This by itself is mutex blocking.

It is not necessarily priority inversion.

2. The LOW, MEDIUM and HIGH Model

To understand priority inversion, imagine three threads:

The important dependency is:

HIGH
  ↓
waits for
  ↓
LOW
  ↓
can be delayed by
  ↓
MEDIUM

This is the classic pattern we want to understand.

3. What Happens Step by Step?

Step 1: LOW acquires the mutex

LOW starts first and successfully executes:

mutex_lock(&my_mutex);

Now LOW owns the mutex.

It enters its critical section and performs some work.

Step 2: HIGH needs the same mutex

HIGH starts and also attempts:

mutex_lock(&my_mutex);

But LOW already owns the mutex.

Therefore HIGH must wait.

At this point:

LOW  -> owns mutex
HIGH -> blocked waiting for mutex

Step 3: MEDIUM starts doing work

MEDIUM is runnable and performs its own work.

The important concept is that MEDIUM does not need the mutex.

Nevertheless, depending on the scheduling configuration, MEDIUM can receive CPU time while LOW is waiting to run.

Step 4: LOW eventually releases the mutex

LOW must eventually execute:

mutex_unlock(&my_mutex);

Only then can HIGH acquire the mutex and continue.

Therefore, HIGH's progress depends indirectly on LOW getting CPU time.

4. Why This Becomes Priority Inversion

Imagine the conceptual priorities are:

HIGH has the highest priority.

However, HIGH is blocked because LOW owns the mutex.

If MEDIUM continues to run while LOW is unable to run and release the mutex, the higher-priority HIGH thread can remain blocked.

This is the surprising part:

A HIGH-priority thread can effectively be delayed because a LOW-priority thread cannot get enough CPU time to release a resource.

That is the essence of priority inversion.

5. Our First Kernel Module: A Simulation

The example module creates three kernel threads:

• low_thread
• medium_thread
• high_thread

The LOW thread acquires a mutex and holds it while simulating long work.

The HIGH thread later attempts to acquire the same mutex.

The MEDIUM thread performs additional work.

The core LOW-thread logic is:

mutex_lock(&my_mutex);

pr_info("[LOW] Mutex Acquired\n");

msleep(10000);

mutex_unlock(&my_mutex);

The HIGH thread later executes:

mutex_lock(&my_mutex);

pr_info("[HIGH] Mutex Acquired\n");

mutex_unlock(&my_mutex);

The result demonstrates the dependency:

LOW owns mutex
        ↓
HIGH needs mutex
        ↓
HIGH waits
        ↓
LOW must eventually release mutex

The complete example is available in the GitHub repository.

6. Important: This Is a Simulation

There is an important distinction to understand.

The initial demonstration uses ordinary kernel threads and a normal mutex. It does not explicitly assign different scheduler priorities to LOW, MEDIUM, and HIGH.

It also does not use rt_mutex to demonstrate real priority inheritance.

Therefore, this first program should be treated as a conceptual priority-inversion simulation, rather than a complete real-time priority-inversion experiment.

The distinction is important:

This makes the simulation useful as a first learning step before studying Linux real-time scheduling and priority inheritance.

7. Why msleep() Is Not CPU-Bound Work

One subtle point is worth emphasizing.

A demonstration using:

msleep(1000);

is sleeping, not continuously consuming CPU.

Therefore, the MEDIUM thread in this particular program does not represent a true CPU-bound medium-priority workload.

A stronger experiment would use controlled CPU work together with explicitly configured scheduling priorities.

That would allow us to demonstrate the scheduler interaction more accurately.

8. The More Complete Experiment

A more realistic experiment can follow this sequence:

1. Create LOW, MEDIUM and HIGH kernel threads.
2. Assign actual scheduling priorities.
3. Use a real-time scheduling policy.
4. Start LOW first.
5. Allow LOW to acquire the mutex.
6. Start HIGH.
7. HIGH attempts to acquire the mutex and blocks.
8. Start MEDIUM.
9. MEDIUM performs CPU-bound work.
10. Observe how long LOW takes to run and release the mutex.
11. Measure HIGH's blocking time.
12. Introduce priority inheritance.
13. Repeat the experiment.
14. Compare the blocking times.

This turns the simple demonstration into a proper scheduling experiment.

9. Priority Inheritance

Priority inheritance is one mechanism used to mitigate priority inversion.

The basic idea is:

If a HIGH-priority thread is waiting for a mutex owned by LOW, LOW can temporarily inherit HIGH's effective priority.

Conceptually:

LOW temporarily receives the higher effective priority.

This makes it more likely that LOW will run, finish its critical section and release the mutex.

Then HIGH can proceed.

10. Without Priority Inheritance

Conceptually:

HIGH waits for LOW
        ↓
LOW remains delayed
        ↓
MEDIUM gets CPU time
        ↓
LOW releases mutex later
        ↓
HIGH waits longer

The important observation is that HIGH's waiting time can be affected by a thread that does not even use the mutex.

11. With Priority Inheritance

With priority inheritance:

HIGH waits for LOW
        ↓
LOW inherits HIGH's effective priority
        ↓
LOW runs sooner
        ↓
LOW releases mutex
        ↓
HIGH continues

The goal is not to make LOW permanently high priority.

The elevated effective priority is associated with the mutex dependency and is temporary.

12. Learning Path

A useful progression for studying this topic is:

Stage 1: Mutex Basics

Start with:

mutex_lock()
mutex_unlock()

Understand:

• mutual exclusion
• critical sections
• ownership
• blocking

Stage 2: Mutex Blocking

Build a small kernel-thread example where one thread owns a mutex and another waits for it.

Stage 3: Priority-Inversion Simulation

Introduce LOW, MEDIUM and HIGH threads.

Understand the dependency:

HIGH → waits for LOW → LOW can be delayed by MEDIUM

Stage 4: Scheduler Interaction

Study:

• Linux scheduler
• task priorities
• real-time scheduling
• SCHED_FIFO
• scheduling behavior

Stage 5: Priority Inheritance

Then move to:

• priority inheritance
• rt_mutex
• real-time locking
• measuring blocking time

This progression makes the real-time concepts much easier to understand.

13. Why Priority Inversion Matters

Priority inversion is particularly important in systems where predictable response time matters.

Examples include:

• embedded systems
• real-time Linux
• industrial control
• robotics
• automotive systems
• telecommunications
• industrial automation
• safety-critical systems

In such systems, simply saying "the HIGH-priority task has the highest priority" is not enough.

If HIGH is blocked on a resource owned by LOW, the actual behavior depends on synchronization and scheduling.

14. From Kernel Scheduling to Cybersecurity

Understanding synchronization and scheduling is also useful when moving toward Linux security and low-level systems work.

For example, a monitoring architecture might contain components such as:

• Linux kernel instrumentation
• eBPF monitoring
• detection logic
• security event collection
• SOC dashboards

The deeper your understanding of kernel scheduling, synchronization and task behavior, the easier it becomes to reason about low-level system behavior rather than treating the kernel as a black box.

15. Key Takeaways

• A mutex protects a shared resource.
• A thread attempting to acquire an owned mutex may block.
• Mutex blocking alone is not necessarily priority inversion.
• Classic priority inversion involves LOW, MEDIUM and HIGH priorities.
• HIGH can wait for LOW because LOW owns the required mutex.
• MEDIUM can indirectly delay HIGH by preventing LOW from progressing.
• The first kernel-module example is a simulation, not a complete real-time scheduler experiment.
• The example does not assign actual LOW/MEDIUM/HIGH scheduler priorities.
• A stronger experiment should use explicit real-time scheduling priorities and controlled CPU-bound work.
• Priority inheritance allows the lower-priority mutex owner to temporarily inherit the effective priority of a higher-priority waiter.
• The objective is to allow the mutex owner to finish its critical section sooner and release the resource.

Conclusion

Priority inversion is a good example of why operating-system performance cannot be understood only by looking at individual thread priorities.

The important question is not simply:

"Which thread has the highest priority?"

We also need to ask:

"Which resource is that thread waiting for, who owns it, and can that owner get enough CPU time to release it?"

Starting with a simple mutex simulation and then progressing toward real-time scheduling and priority inheritance provides a practical way to understand this behavior inside the Linux kernel.

The next logical step is to build Program 6B, using actual scheduling priorities and Linux real-time synchronization mechanisms, and compare HIGH's blocking time without and with priority inheritance.

GitHub Repository

Complete source code and project files:

👉 linux_kernel_priority_inversion

One of the synchronization primitives provided by the Linux kernel is mutex_lock_killable(). It behaves similarly to mutex_lock(), but allows a waiting task to return if interrupted by an appropriate fatal signal.

In this article, we'll explore the fundamentals of mutex_lock_killable() through a simple Linux Kernel Module.

Prerequisites

Before following this tutorial, you should be familiar with:

• Linux Kernel Modules
• C Programming
• Kernel Threads
• Basic Linux Commands
• Synchronization Fundamentals

What is a Mutex?

A Mutex (Mutual Exclusion Lock) ensures that only one execution context can access a shared resource at a time.

Without synchronization:

• Race conditions occur
• Shared data becomes inconsistent
• Kernel behavior becomes unpredictable

Using a mutex guarantees exclusive access to the critical section.

Why mutex_lock_killable()?

The Linux kernel provides multiple mutex APIs.

In many driver scenarios, allowing a blocked task to terminate gracefully is preferable to waiting forever.

Demo Overview

The demonstration module creates two kernel threads.

• Thread-1 acquires the mutex.
• Thread-2 attempts to acquire the same mutex.
• Since the mutex is already held, Thread-2 blocks.
• After Thread-1 releases the mutex, Thread-2 continues execution.

This simple example illustrates how mutual exclusion works inside the Linux kernel.

Core Kernel APIs

The project uses several commonly used kernel APIs.

DEFINE_MUTEX(my_mutex);

mutex_lock_killable(&my_mutex);

mutex_unlock(&my_mutex);

kthread_run(...);

msleep(5000);

pr_info(...);

Each API plays a specific role:

• DEFINE_MUTEX() creates a mutex.
• mutex_lock_killable() acquires the mutex.
• mutex_unlock() releases it.
• kthread_run() creates kernel threads.
• msleep() simulates work.
• pr_info() prints kernel log messages.

Critical Section

A Critical Section is the portion of code that accesses shared resources.

Only one thread should execute this region at any given time.

In the demo:

• Thread-1 enters the critical section.
• Thread-2 waits.
• After unlocking, Thread-2 proceeds.

This guarantees safe access to shared resources.

Building the Module

Compile the module using:

make

If Secure Boot is enabled, sign the kernel module before loading.

sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file \
sha256 \
~/kernel_keys/MOK.key \
~/kernel_keys/MOK.crt \
mutex_killable_demo.ko

Loading the Module

Insert the module:

sudo insmod mutex_killable_demo.ko

Monitor kernel logs:

sudo dmesg -wH

Verify the module:

lsmod | grep mutex_killable_demo

View running kernel threads:

ps -eLf | grep killable

Unload the module:

sudo rmmod mutex_killable_demo

Expected Execution

When the module runs, the output typically follows this sequence:

1. Thread-1 starts.
2. Thread-1 acquires the mutex.
3. Thread-2 starts.
4. Thread-2 blocks while waiting.
5. Thread-1 releases the mutex.
6. Thread-2 acquires the mutex.
7. Thread-2 completes execution.
8. Module unloads successfully.

Practical Use Cases

Understanding kernel synchronization is useful when developing:

• Character Device Drivers
• Platform Drivers
• PCI Drivers
• USB Drivers
• Network Drivers
• Filesystem Modules
• Embedded Linux Systems
• Linux Kernel Subsystems

Key Learning Outcomes

After completing this project, you should understand:

• Linux kernel synchronization
• Kernel threads
• Mutex locking
• Critical sections
• Blocking synchronization
• Kernel logging
• Mutual exclusion
• Safe shared resource access

Important Note

In this educational example, both workers are kernel threads. Since kernel threads typically do not receive user-space signals, the behavior of mutex_lock_killable() appears very similar to mutex_lock().

The primary advantage of mutex_lock_killable() becomes more apparent in kernel drivers where a user-space process blocks while waiting for a mutex and may receive a fatal signal.

Conclusion

Although the demo is intentionally simple, it introduces several core Linux kernel concepts that appear throughout driver development.

Understanding synchronization primitives like mutex_lock_killable() is an important step toward writing reliable kernel modules and avoiding race conditions in concurrent kernel code.

As you continue learning Linux kernel development, you can extend this project by experimenting with additional synchronization mechanisms such as semaphores, spinlocks, completions, wait queues, and reader-writer locks.

GitHub Repository

👉 GitHub Repository:
https://github.com/aj333git/linux_kernel_mutex_killable

Explore the complete source code, build files, and Linux kernel module implementation.

In this project, we build a simple Linux application that demonstrates this architectural pattern using:

• F# as the Control Plane
• C as the Data Plane
• P/Invoke for interoperability
• Linux Shared Libraries (.so)

Although the example is intentionally simple, the same design scales to much larger systems.

Why Separate Control Plane and Data Plane?

Different programming languages excel at different tasks.

In this project:

• F# collects user input.
• The terrain data is stored in a managed array.
• The array is passed directly to native C.
• C prints the terrain grid.

This demonstrates a clean separation of responsibilities.

Project Architecture

The project consists of two independent components.

Control Plane

Responsibilities:

• Read terrain dimensions
• Read elevation values
• Store data in a managed array
• Invoke native code

Technology:

• F#
• .NET 8

Data Plane

Responsibilities:

• Receive terrain data
• Traverse the array
• Print formatted output

Technology:

• C
• GCC
• Shared Library (.so)

Project Structure

terrain_grid/
│
├── control_plane/
│   ├── Program.fs
│   ├── Terrain.fs
│   └── control_plane.fsproj
│
├── data_plane/
│   ├── terrain.c
│   ├── terrain.h
│   └── libterrain.so
│
└── README.md

The Control Plane (F#)

The control plane is responsible for gathering input and forwarding it to the native library.

A simplified version:

let terrain = Array.zeroCreate<double>(rows * cols)

Terrain.display_terrain(
    terrain,
    rows,
    cols)

Notice that the F# code never performs the display itself.

Its only responsibility is orchestration.

Calling Native C with P/Invoke

The bridge between managed and native code is created with DllImport.

[<DllImport("libterrain.so",
    CallingConvention = CallingConvention.Cdecl)>]
extern void display_terrain(
    double[] terrain,
    int rows,
    int cols)

P/Invoke automatically marshals the managed array into a native pointer that the C library can consume.

The Data Plane (C)

The C library receives a pointer to the terrain array and prints the values.

void display_terrain(
    const double* terrain,
    int rows,
    int cols)
{
    printf("%8.2f",
           terrain[i * cols + j]);
}

Because the array is stored contiguously, indexing is straightforward.

Why Use a One-Dimensional Array?

Although the terrain logically represents a matrix, it is stored as a linear array.

Advantages include:

• Contiguous memory
• Better cache locality
• Easier interoperability
• Simpler pointer arithmetic
• Lower overhead

The index calculation is:

index = row * columns + column

This technique is widely used in:

• Scientific computing
• Image processing
• Game engines
• Embedded software
• Numerical libraries

Building the Shared Library

Compile the native library using GCC.

gcc -shared -fPIC terrain.c -o libterrain.so

Building the F# Project

dotnet build

Run:

dotnet run

Sample Output

Rows : 2
Cols : 2

Elevation [0][0] : 2
Elevation [0][1] : 3
Elevation [1][0] : 4
Elevation [1][1] : 5

Terrain Grid (Elevation in meters)
----------------------------------
    2.00    3.00
    4.00    5.00

What This Example Demonstrates

This small application introduces several important concepts.

• F# to C interoperability
• Linux shared libraries
• P/Invoke
• Managed and native memory interaction
• Control Plane / Data Plane architecture
• One-dimensional representation of matrices

Where This Pattern Is Used

The same architecture appears in many production systems.

• Network packet processing
• Linux networking subsystems
• Storage engines
• Embedded firmware
• Robotics
• High-performance computing
• Scientific simulations
• CAD applications
• GIS software
• Image processing pipelines

The complexity may increase, but the architectural principle remains the same.

Possible Extensions

This project can easily be expanded by adding features such as:

• Terrain statistics
• Height normalization
• Gradient calculation
• Heat map generation
• Terrain smoothing
• File import/export
• CSV support
• Binary terrain format
• OpenGL visualization
• GIS integration

Conclusion

This project demonstrates how managed and native languages can work together effectively.

F# provides expressive and concise application logic, while C delivers predictable native execution. By combining the two through P/Invoke, developers can build applications that are both productive and performant.

Although the terrain grid application is intentionally small, it illustrates a design pattern that scales to much larger systems in systems programming, embedded development, and high-performance computing.

GitHub Repository

terrain_grid

https://github.com/aj333git/Civil_Mech_apps/tree/main/terrain_grid

Engineering software often combines multiple programming languages to leverage their individual strengths. A common approach is to implement computational algorithms in native C or C++ while developing the user interface and application logic using higher-level technologies.

This project demonstrates that architecture by building a simple Beam Load Calculator where:

• F# provides the application layer

• Native C performs the numerical computation

• P/Invoke connects both layers

Although the calculation is intentionally simple, the architecture closely resembles that used in professional engineering and scientific applications.

Project Overview

The application computes the total load acting on a beam by summing multiple point loads entered by the user.

The project also introduces:

• Foreign Function Interface (FFI)

• Platform Invocation (P/Invoke)

• Shared libraries (.so)

• Cross-language programming

• Basic engineering computation

Technology Stack

Project Structure

beam_load_calc_

├── beam_load.c
├── beam_load.h
├── beam_load_cal/
│   ├── Program.fs
│   └── beam_load_cal.fsproj
└── README.md

Architecture

The application separates the computational engine from the application layer.

• F# reads user input.

• P/Invoke calls the native library.

• C computes the total beam load.

• F# displays the result.

This separation makes the computational logic reusable and independent of the user interface.

Native C Function

The numerical computation is implemented in C.

double calculate_total_load(const double loads[], int size)
{
    double total = 0.0;

    for (int i = 0; i < size; i++)
        total += loads[i];

    return total;
}

Calling Native Code from F#

The native function is imported using DllImport.

[<DllImport("./beam_load.so")>]
extern double calculate_total_load(double[] loads, int size)

Once imported, it behaves like a normal F# function.

Computer Science Concepts

This project introduces several important software engineering topics.

• Foreign Function Interface (FFI)

• Platform Invocation (P/Invoke)

• Shared Libraries

• Dynamic Linking

• Arrays

• Functional Programming

• Cross-language Programming

• Native Interoperability

• Modular Software Design

Civil and Mechanical Engineering Concepts

From an engineering perspective, the project demonstrates:

• Beam

• Point Load

• Resultant Load

• Load Summation

• Engineering Computation

Although simple, these concepts form the starting point for structural analysis.

Algorithm Analysis

The algorithm visits every load exactly once.

This provides a practical example of applying algorithm analysis to an engineering computation.

Build

Compile the native library.

gcc -shared -fPIC beam_load.c -o beam_load.so

Build the F# application.

cd beam_load_cal
dotnet build

Run the project.

dotnet run

Why This Architecture Matters

Many commercial engineering applications separate the computational engine from the application layer.

Examples include:

• CAD Software

• Structural Analysis Software

• Finite Element Analysis

• Scientific Computing

• Simulation Platforms

• Robotics

• Embedded Systems

Using native libraries for computation improves portability and allows computational code to be reused across different applications.

Future Roadmap

This project serves as a foundation for a larger engineering toolkit.

Planned enhancements include:

• Uniformly Distributed Load (UDL)

• Support Reactions

• Shear Force Diagram

• Bending Moment Diagram

• Beam Deflection

• Material Properties

• Unit Conversion

• PDF Report Generation

• Avalonia Desktop GUI

Key Takeaways

After completing this project, you should understand:

• How F# communicates with native C libraries

• How shared libraries work on Linux

• Basic Platform Invocation (P/Invoke)

• Cross-language software architecture

• Applying programming concepts to engineering problems

Conclusion

This Beam Load Calculator demonstrates how a modern .NET application can integrate with native C code using P/Invoke. While the current implementation focuses on a simple load summation problem, the same architecture can be extended to support more advanced engineering calculations and desktop applications.

By combining F# for application development with C for computational routines, the project illustrates a scalable design that is applicable to engineering software, scientific computing, and high-performance numerical applications.

Source Code

GitHub Repository: 👉 beam_load_calc Explore the complete source code, build files, and module implementation on GitHub.

If you found this project useful, consider giving the repository a ⭐ and sharing your feedback.

In this project, I built a Linux Kernel Loadable Module (LKM) that intentionally creates a deadlock using two kernel threads and two mutexes. The same module also contains a corrected implementation that eliminates the deadlock by enforcing a consistent lock acquisition order.

The objective of this project is educational. It demonstrates not only how deadlocks occur, but also how to debug synchronization issues using kernel logs and how to manage kernel thread lifecycles correctly.

Project Overview

The module supports two execution modes.

Both modes use the same synchronization primitives but differ in the order in which mutexes are acquired.

Technologies Used

• Linux Kernel Modules (LKM)
• Kernel Threads (kthread)
• Mutex Synchronization
• Linux Kernel Logging (pr_info)
• Module Parameters
• dmesg
• Kernel Build System

Understanding the Deadlock

The demonstration creates two worker threads.

The first thread acquires:

LockA

↓

LockB

The second thread acquires:

LockB

↓

LockA

Once each thread acquires its first mutex, both wait forever for the second mutex.

Neither thread can proceed.

This is the classic circular waiting condition that produces a deadlock.

Deadlock Demonstration

The worker thread acquires the first mutex, waits briefly, and then attempts to acquire the second mutex.

Example:

mutex_lock(&lockA);

msleep(2000);

mutex_lock(&lockB);

The second worker performs the opposite operation.

mutex_lock(&lockB);

msleep(2000);

mutex_lock(&lockA);

Running both threads simultaneously causes each thread to wait indefinitely.

Deadlock Prevention

The solution is surprisingly simple.

Both threads acquire the mutexes in exactly the same order.

mutex_lock(&lockA);

mutex_lock(&lockB);

Since every thread follows the same locking sequence, circular waiting is eliminated.

Kernel Threads

The module creates two kernel threads using kthread_run().

Example:

thread1 = kthread_run(worker1, NULL, "thread1");

Each worker performs its task once and then exits.

This project is intentionally kept small so that the synchronization behavior remains easy to observe.

Module Parameters

The module behavior is controlled through a module parameter.

sudo insmod deadlock_demo.ko mode=1

or

sudo insmod deadlock_demo.ko mode=2

This makes it possible to compare the incorrect and corrected implementations without modifying the source code.

Building the Module

Compile the module using the kernel build system.

make

If Secure Boot is enabled, sign the module before loading it.

sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file \
sha256 \
~/kernel_keys/MOK.key \
~/kernel_keys/MOK.crt \
deadlock_demo.ko

Running the Demonstration

Load the deadlock version.

sudo insmod deadlock_demo.ko mode=1

Monitor kernel messages.

dmesg -w

Load the corrected implementation.

sudo insmod deadlock_demo.ko mode=2

Unload the module.

sudo rmmod deadlock_demo

Debugging with dmesg

The Linux kernel provides detailed diagnostic information through dmesg.

Useful commands include:

dmesg | tail -100

dmesg | grep deadlock_demo

Kernel logs make it much easier to understand thread execution, lock acquisition, and module initialization.

Two Important Bugs Encountered

While implementing this demonstration, I encountered two issues that were not caused by the deadlock itself.

1. Incorrect use of kthread_stop()

The worker threads in the corrected implementation naturally terminate after completing their work.

Attempting to stop already-finished threads during module removal resulted in kernel warnings and crashes.

The fix was to avoid calling kthread_stop() for one-shot worker threads.

2. Missing IS_ERR() Validation

Initially, the return value of kthread_run() was not validated.

Proper kernel programming requires checking whether thread creation failed.

Example:

if (IS_ERR(thread1))

Adding this validation makes the module more robust and avoids invalid pointer usage.

A detailed analysis of both debugging sessions will be presented in a future article.

Key Concepts Covered

• Linux Kernel Modules
• Kernel Threads
• Mutexes
• Mutual Exclusion
• Critical Sections
• Deadlocks
• Deadlock Prevention
• Lock Ordering
• Module Parameters
• Kernel Logging
• Thread Lifecycle
• Basic Kernel Debugging

What I have Learned

By studying this project,I have gained hands-on experience with:

• Writing Linux Kernel Modules
• Creating kernel threads
• Synchronizing shared resources
• Understanding why deadlocks occur
• Preventing deadlocks using proper lock ordering
• Reading kernel logs
• Debugging synchronization problems

Future Articles

This project is the first step in a larger Linux kernel synchronization series.

Upcoming articles will provide a deeper discussion of:

• Complete source code walkthrough
• Thread lifecycle management
• kthread_run() internals
• kthread_stop() best practices
• IS_ERR(), ERR_PTR(), and PTR_ERR()
• Linux kernel debugging techniques
• Kernel call trace analysis
• Synchronization best practices

Conclusion

Deadlocks are easier to understand when they can be reproduced in a controlled environment.

This Linux Kernel Module demonstrates both an incorrect synchronization strategy and its corrected implementation using only two kernel threads and two mutexes. Although the project is intentionally small, it illustrates several important concepts that frequently appear in operating system design and Linux kernel development.

Understanding synchronization primitives at this level provides a solid foundation for exploring more advanced topics such as semaphores, read-write locks, completions, wait queues, workqueues, RCU, and lock-free programming.

GitHub Repository

Source code for this project is available at:

https://github.com/aj333git/linux_kernel_deadlock_demo

Recently, an interesting Linux Kernel Mailing List (LKML) discussion initiated by Yury Norov explored different approaches for region allocation, comparing Bitmap, IDA, and Maple Tree. Inspired by that discussion, I built a small Linux Kernel Module (LKM) to understand how Bitmap and IDA work internally and to benchmark their allocation time.

This article walks through the concepts, implementation, and practical differences between these allocators.

Why This Topic Matters

Kernel developers frequently face questions such as:

• Should a resource be represented as individual IDs?
• Should contiguous regions be allocated?
• Which data structure scales better as the number of resources grows?
• How much overhead does each allocator introduce?

The answer depends entirely on the allocation problem.

Motivation

The implementation was inspired by the following LKML discussion:

Message-ID

20260717053241.916441-1-ynorov@nvidia.com

The discussion compares several allocation mechanisms for variable-sized regions, including:

• Bitmap
• IDA
• Maple Tree

Although this module implements only Bitmap and IDA, understanding these two allocators provides an excellent foundation before exploring Maple Tree.

Understanding Bitmap Allocation

A bitmap represents every resource using a single bit.

Each bit has only two possible states:

• 0 → Free
• 1 → Allocated

When a contiguous region is required, the kernel searches for consecutive zero bits and marks them as allocated.

This approach is extremely memory efficient because one bit represents one resource.

Common Kernel APIs

bitmap_zalloc()

bitmap_find_next_zero_area()

bitmap_set()

bitmap_clear()

bitmap_free()

Bitmap Allocation Workflow

1. Allocate bitmap memory.
2. Search for the next free contiguous region.
3. Mark the bits as allocated.
4. Perform work.
5. Clear the bits.
6. Free the bitmap.

Understanding IDA

IDA stands for Integer ID Allocator.

Unlike Bitmap, IDA does not allocate contiguous regions.

Instead, it allocates unique integer identifiers.

Typical output might be:

0
1
2
3
4
5

Each allocation is completely independent.

IDA is widely used throughout the Linux kernel whenever objects require unique identifiers.

Common Kernel APIs

ida_init()

ida_alloc()

ida_free()

ida_destroy()

IDA Allocation Workflow

1. Initialize IDA.
2. Allocate one integer ID.
3. Use the ID.
4. Return the ID.
5. Destroy the allocator.

Why Bitmap and IDA Solve Different Problems

Although both allocate resources, they are designed for different scenarios.

Bitmap focuses on contiguous regions, while IDA focuses on unique identifiers.

Examples include:

Bitmap

• CPU masks
• Memory regions
• DMA regions
• Block allocation
• Fixed resource maps

IDA

• Device numbers
• Driver IDs
• Kernel object identifiers
• Subsystem IDs

Linux Kernel Module Overview

The module supports three execution modes.

Mode 1

Runs only the Bitmap allocator.

sudo insmod region_demo.ko mode=1

Mode 2

Runs only the IDA allocator.

sudo insmod region_demo.ko mode=2

Mode 3

Runs both allocators for comparison.

sudo insmod region_demo.ko mode=3

Measuring Allocation Time

The module measures allocation latency using the kernel timing API.

start = ktime_get();

/* allocation */

end = ktime_get();

ktime_to_ns(end - start);

Although the benchmark is intentionally simple, it demonstrates how kernel operations can be measured at nanosecond resolution.

Example Kernel Output

Bitmap

Region allocator demo loaded

Bitmap allocated region

start=0 size=8

Bitmap alloc time 13800 ns

IDA

IDA allocated id=0

IDA allocated id=1

IDA allocated id=2

...

IDA alloc time 19200 ns

Bitmap vs IDA

Where Does Maple Tree Fit?

Modern Linux kernels increasingly use Maple Tree for managing variable-sized ranges efficiently.

Compared with Bitmap and IDA, Maple Tree offers:

• Better scalability
• Efficient range management
• Reduced tree overhead
• Excellent performance for large sparse address spaces

Many newer kernel subsystems are gradually adopting Maple Tree where traditional tree-based structures were previously used.

Key Takeaways

• Bitmap is ideal for contiguous resource allocation.
• IDA is designed for allocating unique integer identifiers.
• Both allocators solve different problems and are widely used throughout the Linux kernel.
• Measuring allocator performance helps understand the trade-offs between different kernel data structures.
• Learning Bitmap and IDA provides an excellent foundation before exploring Maple Tree and more advanced kernel memory management techniques.

Conclusion

Kernel allocation is much more than simply requesting memory. Different kernel subsystems require different allocation strategies depending on whether they manage contiguous regions, sparse identifiers, or large variable-sized ranges.

This small Linux Kernel Module demonstrates how Bitmap and IDA operate internally, provides a simple timing benchmark, and connects practical experimentation with an ongoing LKML discussion. If you're learning Linux kernel development, implementing these allocators yourself is one of the best ways to understand the design decisions behind modern kernel resource management.

Source Code

GitHub Repository

https://github.com/aj333git/linux_kernel_bitmap_ida

References

• Linux Kernel Documentation
• Linux Kernel Bitmap API
• Linux Kernel IDA API
• Linux Kernel Mailing List (LKML)
• LKML Message-ID: 20260717053241.916441-1-ynorov@nvidia.com

In this article, I will build a Linux Kernel Module (LKM) that exposes scheduler information through the /proc filesystem, allowing us to inspect processes directly from kernel space.

The module targets Ubuntu 22.04 (Linux Kernel 5.15.x) and uses only modern kernel APIs.

What I will Learn

By the end of this article I will understand:

• Linux scheduler fundamentals
• What task_struct represents
• The current process pointer
• Process priorities
• Nice values
• Process states
• CPU assignment
• The /proc filesystem
• The seq_file interface
• Safe kernel reads using READ_ONCE()
• Process iteration using for_each_process()

Why Build This Project?

Most Linux users interact with commands like:

ps
top
htop

These utilities obtain process information from the kernel.

Instead of only using these tools, this project demonstrates how the kernel itself provides scheduler information.

This makes it an excellent project for learning:

• Linux Kernel Programming
• Operating Systems
• Device Driver Development
• Embedded Linux
• Linux Internals

Project Overview

The kernel module creates a new file:

/proc/scheduler_demo

Reading this file displays scheduler information about every process currently running on the system.

Example:

cat /proc/scheduler_demo

How the Module Works

The execution flow is straightforward:

1. Load the kernel module.
2. Register a new /proc entry.
3. The user reads the file.
4. The kernel invokes a callback.
5. Process information is collected.
6. Results are returned to userspace.

Building Blocks Used

Understanding task_struct

Every process in Linux is represented by a task_struct.

It contains information such as:

• Process ID
• Process name
• Scheduling priority
• Nice value
• Current CPU
• Memory information
• Parent and child relationships
• Process state
• Credentials
• Scheduling class

Nearly every scheduler-related API operates on a task_struct.

The current Pointer

Linux always knows which process is currently executing.

That process is accessible through:

current

From this pointer we can obtain information such as:

• PID
• Process name
• Priority
• Nice value
• Current scheduler state

Enumerating Every Process

The kernel provides an iterator for traversing every process.

for_each_process(task)
{
    /* Scheduler information */
}

This macro walks through the kernel's process list and allows the module to inspect each task.

Reading the Current CPU

Every process executes on a CPU core.

The kernel provides:

task_cpu(task)

which returns the CPU currently associated with that task.

This is useful when studying:

• SMP systems
• CPU scheduling
• Load balancing
• Processor affinity

Understanding Priorities

Linux scheduling uses several priority values.

This project prints:

• Scheduler priority
• Static priority
• Normal priority
• Nice value

Together these values determine how the scheduler treats a process.

Nice Values

User processes can influence scheduling through the nice value.

Typical range:

The module retrieves it using:

task_nice(task)

Process States

Processes constantly transition between states.

Examples include:

• Running
• Runnable
• Sleeping
• Stopped
• Zombie

The module safely reads the scheduler state using:

READ_ONCE(task->__state)

Using READ_ONCE() prevents compiler optimizations from producing inconsistent reads when scheduler data changes concurrently.

Detecting Runnable Tasks

The module checks whether a process is runnable using:

task_is_running(task)

This modern helper is preferred over manually interpreting scheduler state bits.

Why Use /proc?

The /proc filesystem is designed to expose runtime kernel and process information.

Examples include:

/proc/cpuinfo
/proc/meminfo
/proc/modules
/proc/uptime

Our module simply adds another entry:

/proc/scheduler_demo

Why seq_file?

Kernel output can become very large.

Instead of manually managing buffers, Linux provides the seq_file interface.

Benefits include:

• Automatic buffering
• Safe iteration
• Large output support
• Simpler implementation
• Better maintainability

The module prints information using:

seq_printf(...)

Creating the /proc Entry

The module creates the file using:

proc_create(...)

When the module is unloaded, it removes the entry using:

remove_proc_entry(...)

This keeps the filesystem clean and avoids leaving stale entries behind.

Kernel Module Lifecycle

Every Linux Kernel Module follows a lifecycle.

Initialization:

module_init(...)

Cleanup:

module_exit(...)

Initialization registers resources, while cleanup releases them before the module is unloaded.

APIs Covered

Key Takeaways

This project demonstrates several important Linux kernel concepts in a compact and practical example.

You learned how to:

• Create a Linux Kernel Module
• Add custom entries to the /proc filesystem
• Traverse every process in the kernel
• Read scheduler metadata
• Display process priorities and nice values
• Access CPU information
• Safely inspect process states
• Generate formatted kernel output using seq_file
• Follow modern Linux Kernel 5.15 programming practices

Although intentionally simple, this module forms a strong foundation for studying advanced scheduler topics such as the Completely Fair Scheduler (CFS), scheduling classes, CPU affinity, load balancing, kernel threads, and scheduler internals.

Conclusion

The Linux scheduler is responsible for deciding which process runs, when it runs, and on which CPU. By exposing scheduler information through a custom /proc entry, this project provides a practical way to explore those internals without modifying the kernel itself.

For learning Linux kernel programming, operating systems, embedded Linux, or device driver development, implementing small projects like this is one of the most effective ways to understand how the kernel works beneath the surface.

GitHub Repository: 👉 linux_kernel_scheduler Explore the complete source code, build files, and module implementation on GitHub.

Concurrency is a fundamental aspect of Linux kernel programming. Device drivers often execute in multiple contexts, making synchronization essential whenever shared data is accessed.

One of the most widely used synchronization primitives in the Linux kernel is the mutex. A mutex ensures that only one execution context can access a shared resource at a time, preventing race conditions and preserving data consistency.

In this article, I will build a mental model around Linux mutexes using a small driver that demonstrates how multiple kernel threads safely share a common driver context.

Why Do Device Drivers Need Synchronization?

Modern Linux systems execute many threads concurrently.

Imagine a network driver where one thread transmits packets while another receives packets and a third periodically gathers statistics.

All of them access the same driver state.

Without synchronization, concurrent updates can corrupt shared data.

Typical shared driver state includes:

• Packet counters

• Device status

• Buffers

• Hardware configuration

• Runtime statistics

Whenever multiple execution contexts access shared mutable data, synchronization becomes necessary.

Driver Context

A common Linux driver design pattern is to store all runtime information inside a single structure called the Driver Context.

Instead of scattering global variables throughout the driver, everything is encapsulated into one object.

Example:

struct driver_context {
    struct mutex lock;
    int tx_packets;
    int rx_packets;
    bool device_enabled;
    char device_name[32];
};

This approach improves:

• Maintainability

• Readability

• Encapsulation

• Synchronization

• Scalability

Nearly every production-quality Linux driver maintains some form of private context structure.

Shared Writable Data

Inside the driver context are variables that multiple threads may both read and modify.

Examples include:

• tx_packets

• rx_packets

• device_enabled

These variables are called Shared Writable Data because:

• multiple threads access them

• their values change during execution

Since concurrent access is possible, synchronization is required.

Critical Sections

A Critical Section is the portion of code that accesses shared writable data.

Only one execution context should execute a critical section at any given time.

A mutex creates this protected region.

Example:

mutex_lock(&ctx->lock);

ctx->tx_packets++;

mutex_unlock(&ctx->lock);

Everything between mutex_lock() and mutex_unlock() belongs to the critical section.

Linux Mutex APIs

The Linux kernel provides a small but powerful mutex API.

mutex_init()

Initializes a mutex before it can be used.

mutex_init(&ctx->lock);

mutex_lock()

Acquires exclusive ownership of the mutex.

If another thread already owns it, the caller sleeps until the mutex becomes available.

mutex_lock(&ctx->lock);

mutex_lock_interruptible()

Works similarly to mutex_lock(), but allows the waiting task to be interrupted by a signal.

if (mutex_lock_interruptible(&ctx->lock))
    return -EINTR;

mutex_unlock()

Releases the mutex so another waiting thread can proceed.

mutex_unlock(&ctx->lock);

mutex_destroy()

Destroys a dynamically initialized mutex during cleanup.

mutex_destroy(&ctx->lock);

Worker Threads

The demo launches three kernel threads.

TX Thread

Responsible for transmitting packets.

Responsibilities include:

• Acquire mutex

• Update TX counter

• Release mutex

RX Thread

Responsible for receiving packets.

It demonstrates the use of mutex_lock_interruptible() while updating the receive counter.

Statistics Thread

Periodically reads the driver state.

Although it performs only reads, it still acquires the mutex because the shared data may be changing concurrently.

Why Lock During Reads?

Many beginners assume only writers require synchronization.

This is incorrect.

Suppose the TX thread updates a packet counter while the Statistics thread is reading it.

Without synchronization, the reader may observe inconsistent state.

Production Linux drivers therefore commonly synchronize both readers and writers whenever they access shared mutable data.

Race Conditions

A race condition occurs whenever multiple execution contexts modify shared data without proper synchronization.

Consider a shared counter with an initial value of 5.

Two CPUs increment it simultaneously.

Each CPU reads the value 5, increments it, and writes back 6.

The expected value is 7, but the final value becomes 6.

One increment is lost.

This classic problem is known as a Race Condition.

Mutexes prevent this by allowing only one thread to execute the critical section at a time.

Lock Contention

Lock contention occurs when multiple threads attempt to acquire the same mutex simultaneously.

Only one thread can own the mutex.

The remaining threads wait until it becomes available.

Some contention is expected in concurrent systems.

However, excessive contention reduces:

• Throughput

• Parallelism

• Overall performance

Sleeping While Holding a Mutex

One important characteristic of Linux mutexes is that the owner is allowed to sleep while holding the lock.

For demonstration purposes, the TX thread intentionally sleeps after acquiring the mutex.

This makes lock contention easy to observe because other threads remain blocked while the mutex is held.

Although this behavior is useful for teaching synchronization, production drivers generally avoid holding mutexes longer than necessary.

Production Driver vs Demonstration Driver

The objective of this project is education rather than performance.

The demonstration intentionally keeps the mutex for a longer duration so waiting threads can be observed.

A production-quality driver typically follows a different philosophy.

Keep critical sections:

• Small

• Predictable

• Efficient

Protect only the shared resource.

Release the mutex immediately after the shared state has been updated.

Long critical sections increase contention and reduce concurrency.

Key Concepts

Interview Takeaways

If asked about mutexes during a Linux kernel interview, remember these points:

• A mutex protects shared writable data.

• Only one thread may own a mutex at a time.

• Mutexes are sleeping locks.

• mutex_lock_interruptible() allows interrupted waits.

• Critical sections should be kept as short as possible.

• Driver contexts encapsulate shared runtime state.

• Production drivers synchronize both readers and writers.

Conclusion

The purpose of this mini driver is not to demonstrate multithreading itself.

The worker threads simply create concurrent access.

The real lesson is understanding how a single Linux mutex protects a shared driver context, ensuring safe access to shared mutable data while preventing race conditions.

Once these core ideas become clear, you'll recognize the same synchronization pattern throughout the Linux kernel and in production-quality device drivers.

GitHub Repository: 👉 linux_kernel_mutex4 Explore the complete source code, build files, and module implementation on GitHub.

One of the most widely used synchronization primitives in Linux is the mutex. Unlike spinlocks, mutexes are sleeping locks, making them ideal for protecting shared resources in process context where blocking is allowed.

In this article, I'll build a Linux kernel module that demonstrates mutex synchronization using two competing kernel threads while exploring the design principles behind mutexes.

Why Do We Need Mutexes?

Consider two kernel threads incrementing the same shared counter.

Without synchronization, both threads may read the same value before either writes it back, causing one update to be lost.

This classic race condition leads to inconsistent program behavior.

A mutex ensures that only one thread can enter the critical section at a time, preserving data integrity.

Project Overview

The module demonstrates:

• Linux kernel threads (kthread_run())

• Shared resource protection

• Dynamic memory allocation

• Static mutex initialization

• Dynamic mutex initialization

• Critical sections

• Safe synchronization

• Module initialization and cleanup

The shared driver context contains both the protected data and the mutex.

struct demo_context {
    int shared_counter;
    struct mutex lock;
};

The counter is accessed by two independent kernel threads, both competing for the same mutex.

Static vs Dynamic Mutex Initialization

Linux provides two ways to initialize a mutex.

Static Initialization

Static mutexes are initialized automatically before the module starts executing.

static DEFINE_MUTEX(global_mutex);

This approach is ideal for global or module-wide locks whose lifetime matches the module.

Advantages

• No initialization call required

• Simple and efficient

• Lifetime equals module lifetime

Dynamic Initialization

When mutexes are embedded inside dynamically allocated objects, they must be initialized explicitly.

ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);

mutex_init(&ctx->lock);

This is the preferred approach for driver-specific objects allocated at runtime.

Advantages

• Works with dynamically allocated structures

• Common in device drivers

• Flexible for multiple device instances

Creating Kernel Threads

The module launches two worker threads.

thread1 = kthread_run(worker, (void *)1, "mutex_thread1");

thread2 = kthread_run(worker, (void *)2, "mutex_thread2");

Both execute the same worker function and continuously compete for the mutex.

Entering the Critical Section

The critical section begins when the mutex is acquired.

mutex_lock(&ctx->lock);

ctx->shared_counter++;

mutex_unlock(&ctx->lock);

If another thread already owns the mutex, the current thread is automatically put to sleep until the lock becomes available.

Unlike busy waiting, sleeping conserves CPU resources and improves overall system efficiency.

Sleeping Inside a Mutex

One interesting aspect of this example is that the worker intentionally sleeps while holding the mutex.

msleep(1000);

This demonstrates an important property of Linux mutexes:

• Sleeping is allowed while holding a mutex.

• Sleeping is forbidden while holding a spinlock.

Although sleeping inside a mutex is legal, real production code should keep critical sections as short as possible to minimize lock contention.

Understanding the Worker Thread

Each worker repeatedly performs the following sequence:

1. Acquire the mutex.

2. Increment the shared counter.

3. Print the updated value.

4. Sleep briefly.

5. Release the mutex.

6. Sleep outside the critical section.

This repeated competition clearly demonstrates serialized access to shared data.

Expected Kernel Output

A typical execution looks like this:

===== Mutex Demo Loaded =====

Static mutex locked once

Thread 1 acquired mutex
Thread 1 Counter=1
Thread 1 releasing mutex

Thread 2 acquired mutex
Thread 2 Counter=2
Thread 2 releasing mutex

Notice that the counter always increases sequentially.

There is never simultaneous access to the protected resource.

Important Mutex APIs

Essential Mutex Rules

1. Only One Owner

Only one task may own a mutex at any moment.

Any additional thread attempting to acquire it will sleep until it becomes available.

2. Owner Must Unlock

The thread that acquires the mutex must also release it.

Unlocking a mutex from another thread leads to undefined behavior.

3. Recursive Locking Is Not Allowed

Attempting to acquire the same mutex twice from the same thread causes a deadlock.

Avoid recursive locking unless a different synchronization primitive is specifically designed for it.

4. Mutexes May Sleep

Because mutex_lock() may block, mutexes are valid only in process context.

They are commonly used inside:

• Kernel threads

• System calls

• Character device operations

• Driver read/write methods

5. Never Use Mutexes in Atomic Context

Mutexes should never be used inside:

• Interrupt handlers

• SoftIRQs

• Tasklets

• Timers

• Other atomic contexts

Blocking is not permitted in these execution contexts.

6. Keep Critical Sections Small

The protected region should contain only the minimum required operations.

Smaller critical sections improve scalability, reduce contention, and increase overall system responsiveness.

7. Follow Consistent Lock Ordering

When multiple locks are required, always acquire them in the same order throughout the codebase.

Consistent ordering is one of the simplest and most effective techniques for preventing deadlocks.

Module Lifecycle

Module Initialization

During module loading, the following sequence occurs:

• Allocate driver context

• Initialize dynamic mutex

• Demonstrate static mutex

• Launch two kernel threads

Module Cleanup

During module removal:

• Stop both kernel threads

• Free allocated memory

• Exit cleanly

Proper cleanup ensures that no kernel resources are leaked.

Where Are Mutexes Used?

Mutexes appear throughout the Linux kernel and are commonly used in:

• Character device drivers

• Platform drivers

• USB drivers

• PCI drivers

• I2C drivers

• SPI drivers

• Filesystems

• Network drivers

• Virtual device drivers

• Embedded Linux systems

Typical protected resources include:

• Device state

• Hardware registers

• Linked lists

• Queues

• Buffers

• Configuration structures

• Statistics

• Driver context objects

Final Thoughts

Mutexes are among the most important synchronization primitives in Linux kernel development. Understanding when to use them—and equally important, when not to use them—is fundamental for writing reliable kernel modules and production-quality device drivers.

This demonstration illustrates the complete lifecycle of mutex usage, from initialization and lock acquisition to protecting shared resources and performing clean module shutdown. Once these fundamentals are mastered, I can confidently move on to advanced synchronization mechanisms such as semaphores, completions, wait queues, spinlocks, reader-writer locks, RCU, and lock-free programming techniques.

GitHub Repository: 👉 linux_kernel_mutex3 Explore the complete source code, build files, and module implementation on GitHub.

A hands-on Linux Kernel Module (LKM) demonstrating kernel mutexes, kernel threads, critical sections, thread synchronization, and proper module lifecycle management.

Table of Contents

• Overview
• Learning Objectives
• Concepts Covered
• Project Structure
• Build Requirements
• Compilation
• Module Signing (Secure Boot)
• Loading the Module
• Viewing Kernel Logs
• Unloading the Module
• Execution Flow
• Program Architecture
• Source Code Walkthrough
• Kernel Threads
• Driver Private Context
• Understanding Mutex
• Critical Section
• mutex_lock()
• mutex_lock_interruptible()
• Thread Competition
• Expected Execution Timeline
• Kernel Log Explanation
• Module Cleanup
• Important Kernel APIs
• Why Mutex Instead of Spinlock?
• Common Interview Questions
• Exercises
• References

Overview

This project demonstrates one of the most fundamental synchronization primitives in the Linux kernel:

• struct mutex

The module creates two kernel threads that continuously compete for a shared mutex while modifying a shared variable.

The program intentionally introduces contention so the kernel scheduler blocks one thread while the other owns the mutex.

This is exactly how real Linux device drivers protect shared resources.

Examples include

• USB Drivers
• PCI Drivers
• Character Drivers
• I2C Drivers
• SPI Drivers
• Network Drivers
• Filesystem Drivers

Learning Objectives

After completing this example you should understand

• Linux Kernel Modules
• Kernel Threads
• Driver Private Data
• Shared Resources
• Race Conditions
• Critical Sections
• Sleeping Locks
• Mutex Initialization
• Mutex Acquisition
• Mutex Release
• Interruptible Waiting
• Thread Scheduling
• Proper Cleanup

Concepts Covered

Project Structure

mutex_demo/
│
├── Makefile
├── mutex_demo.c
└── README.md

Build Requirements

• Linux Kernel Headers
• GCC
• Make
• Root Privileges
• Secure Boot Keys (if Secure Boot enabled)

High Level Execution Flow

           insmod
              │
              ▼
     module_init()
              │
              ▼
     Initialize Mutex
              │
              ▼
      shared_counter=0
              │
              ▼
    Create Kernel Thread T1
              │
              ▼
    Create Kernel Thread T2
              │
              ▼
   Both compete for mutex
              │
              ▼
 Enter Critical Section
              │
              ▼
 Increment Counter
              │
              ▼
 Release Mutex
              │
              ▼
 Repeat
              │
              ▼
         rmmod
              │
              ▼
     Stop Both Threads
              │
              ▼
     Destroy Mutex
              │
              ▼
       Module Exit

Program Architecture

                    +------------------------+
                    | Linux Kernel Module    |
                    +-----------+------------+
                                |
                                |
                    +-----------v------------+
                    | Driver Private Context |
                    +------------------------+
                    | shared_counter         |
                    | struct mutex mymtx     |
                    +-----------+------------+
                                |
          +---------------------+---------------------+
          |                                           |
          |                                           |
+---------v---------+                     +-----------v----------+
| Kernel Thread T1  |                     | Kernel Thread T2     |
| mutex_lock()      |                     | mutex_lock_interruptible() |
+---------+---------+                     +-----------+----------+
          |                                           |
          +---------------------+---------------------+
                                |
                                ▼
                        Shared Counter

Driver Private Context

struct mydrv_priv
{
    int shared_counter;
    struct mutex mymtx;
};

This structure stores

• Driver state
• Shared resources
• Synchronization objects

Nearly every Linux device driver maintains a private structure similar to this.

Shared Counter

drvctx.shared_counter++;

Both threads modify this variable.

Without synchronization

T1 reads 10

T2 reads 10

T1 writes 11

T2 writes 11

Expected

12

Actual

11

This is called a

Race Condition

Critical Section

A critical section is any code accessing shared resources.

mutex_lock()

↓

Modify Shared Data

↓

mutex_unlock()

In this example

mutex_lock(&drvctx.mymtx);

drvctx.shared_counter++;

mutex_unlock(&drvctx.mymtx);

Only one thread may execute this block at any time.

Kernel Threads

The module creates two kernel threads.

Thread 1

↓

worker_lock()

Thread 2

↓

worker_interruptible()

Creation

kthread_run()

Stopping

kthread_stop()

Thread loop

while (!kthread_should_stop())

This is the standard Linux kernel thread pattern.

Understanding Mutex

Mutex means

Mutual Exclusion

Only one thread may own the mutex.

          Mutex

      Locked
         │
         ▼

      Thread 1

Thread 2 waits

Thread 3 waits

Once unlocked

Scheduler wakes

↓

Next waiting thread

mutex_init()

mutex_init(&drvctx.mymtx);

Initializes the mutex.

Internally

Unlocked

Owner = NULL

Wait Queue = Empty

mutex_lock()

mutex_lock(&drvctx.mymtx);

If mutex is free

Acquire immediately

If mutex is busy

Sleep

↓

Scheduler switches CPU

↓

Wake later

This is called a sleeping lock.

mutex_unlock()

mutex_unlock(&drvctx.mymtx);

Releases ownership.

If another thread is waiting

Wake waiting thread

↓

Scheduler

↓

Acquire mutex

mutex_lock_interruptible()

ret = mutex_lock_interruptible(&drvctx.mymtx);

Unlike

mutex_lock()

this version may return early if interrupted.

Return value

0

↓

Lock acquired

Non-zero

Interrupted while waiting

Program

if (ret)
{
    printk(...);
    continue;
}

Thread Competition

Time

T1 -------------------- LOCK ------------------- UNLOCK

                    T2 waiting

                              LOCK

                              UNLOCK

T1 waiting

LOCK

UNLOCK

Only one thread executes inside the critical section.

Expected Timeline

Counter

0

Thread 1

Acquire Mutex

Counter=0

Increment

Counter=1

Sleep

Unlock

Thread 2

Waiting...

Acquire

Counter=1

Increment

Counter=2

Unlock

Thread 1

Acquire

Counter=2

Increment

Counter=3

and so on.

Expected Kernel Log

Mutex Demo: Module Loaded

Mutex initialized

T1: waiting for mutex

T1: acquired mutex, counter=0

T2: waiting for mutex (interruptible)

T1: releasing mutex, counter=1

T2: acquired mutex, counter=1

T2: releasing mutex, counter=2

T1: acquired mutex, counter=2

T1: releasing mutex, counter=3

...

Observe

• No overlapping critical sections
• Counter always increases correctly
• Mutex serializes access

Module Cleanup

rmmod

↓

module_exit()

↓

kthread_stop(T1)

↓

kthread_stop(T2)

↓

mutex_destroy()

↓

Exit

mutex_destroy()

mutex_destroy(&drvctx.mymtx);

Normally performs little work.

Useful when

CONFIG_DEBUG_MUTEXES=y

Kernel debug builds perform additional consistency checks.

Important Kernel APIs

Why Use Mutex?

Advantages

• Simple API
• Prevents race conditions
• Sleeping lock
• Scheduler friendly
• No busy waiting
• Excellent for long critical sections

Ideal for

• Device Drivers
• Filesystems
• Networking
• USB
• Character Drivers

Mutex vs Spinlock

Real Driver Examples

Mutexes commonly protect

Device Registers

Configuration Structures

Driver State

Shared Buffers

Reference Counters

Linked Lists

Open File State

Device Queues

Key Takeaways

This project demonstrates

• Linux Kernel Module lifecycle
• Driver private context
• Shared resource protection
• Mutex initialization
• Mutex acquisition
• Mutex release
• Interruptible mutex locking
• Kernel thread creation
• Thread synchronization
• Scheduler interaction
• Critical section protection
• Race condition avoidance
• Safe module cleanup

Although intentionally simple, the synchronization pattern used here is the same pattern employed throughout production Linux kernel drivers.

Summary

This Linux Kernel Module serves as a compact yet practical introduction to sleeping synchronization in the Linux kernel. By creating two competing kernel threads that protect a shared resource with a mutex, it demonstrates the complete lifecycle of mutex usage—from initialization and acquisition to release and cleanup—while reinforcing essential concepts such as race conditions, critical sections, scheduler interaction, and proper kernel module design. It provides a solid foundation before progressing to more advanced synchronization primitives such as spinlocks, semaphores, completions, wait queues, reader-writer locks, and Read-Copy-Update (RCU).

GitHub Repository: 👉 linux_kernel_mutex Explore the complete source code, build files, and module implementation on GitHub.

Concurrency is everywhere inside the Linux kernel. Multiple threads, processes, softirqs, tasklets, and interrupt handlers can access the same shared data simultaneously.

Without synchronization, this leads to:

• Race conditions

• Data corruption

• Unpredictable behavior

• Difficult-to-debug kernel crashes

Two of the most common synchronization primitives used by kernel developers are:

• Mutex

• Spinlock

Although both protect critical sections, they work very differently internally and are designed for different situations.

The Locking Problem

Imagine three kernel threads attempting to update the same shared counter:

shared_counter++;

If all threads execute this simultaneously, the result becomes unpredictable.

To avoid this, we protect the critical section:

lock();

shared_counter++;

unlock();

Only one thread can enter the protected region at a time.

The important question is:

What happens to the threads that fail to acquire the lock?

The answer depends on whether we're using a mutex or a spinlock.

How a Mutex Works

A mutex allows only one owner.

When a thread attempts to acquire a locked mutex:

1. It cannot proceed.

2. It is put to sleep.

3. The scheduler removes it from the CPU.

4. It waits until the mutex becomes available.

Example:

mutex_lock(&my_mutex);

/* critical section */

mutex_unlock(&my_mutex);

Characteristics

• Blocking lock

• Sleeping is allowed

• Context switch occurs

• Suitable for long critical sections

How a Spinlock Works

A spinlock behaves differently.

If a thread cannot acquire the lock:

• It does not sleep

• It repeatedly checks whether the lock is available

• It continuously waits ("spins")

Example:

spin_lock(&my_lock);

/* critical section */

spin_unlock(&my_lock);

Conceptually:

while (lock_is_busy)
    ;

Real Linux implementations are much more optimized than this simple representation.

Why Spinning Can Be Faster

Putting a thread to sleep is expensive.

The kernel must:

1. Invoke the scheduler

2. Save thread state

3. Perform a context switch

4. Wake the thread later

5. Perform another context switch

A mutex therefore incurs at least:

Sleep Context Switch
+
Wakeup Context Switch

For very short critical sections, this overhead can exceed the actual work being protected.

The Performance Rule

Let:

• t_locked = time spent in critical section

• t_ctxsw = context switch time

If:

t_locked < 2 × t_ctxsw

then using a mutex becomes inefficient.

The system spends more time managing threads than performing useful work.

This phenomenon is known as thrashing.

In such cases, a spinlock is often the better choice.

Mutex vs Spinlock Comparison

Atomic Context Changes Everything

The Linux kernel has a very important rule:

Code running in atomic context cannot sleep.

Examples include:

• Interrupt handlers

• Softirqs

• Tasklets

• Other atomic execution paths

Since mutexes sleep internally, they cannot be used here.

Incorrect:

irq_handler()
{
    mutex_lock(&lock);   /* WRONG */
}

Correct:

irq_handler()
{
    spin_lock(&lock);

    /* critical section */

    spin_unlock(&lock);
}

Sleeping Inside a Spinlock Is Forbidden

This is a common beginner mistake.

Never do:

spin_lock(&lock);

msleep(100);

spin_unlock(&lock);

Or:

spin_lock(&lock);

wait_event(...);

spin_unlock(&lock);

A spinlock assumes the holder will release it quickly.

Sleeping while holding a spinlock can freeze the system or trigger kernel warnings.

Practical Decision Guide

Use a Spinlock When

• Running in interrupt context

• Running in atomic context

• Critical section is extremely short

• Sleeping is not allowed

• Maximum performance is required

Example:

spin_lock(&lock);
counter++;
spin_unlock(&lock);

Use a Mutex When

• Running in process context

• Blocking I/O may occur

• Sleeping may occur

• Critical section is relatively long

Example:

mutex_lock(&lock);

copy_to_user(...);

mutex_unlock(&lock);

Determining the Current Context

Linux provides helpers to determine where code is executing.

Example:

if (in_task())
{
    /* process context */
}
else
{
    /* atomic or interrupt context */
}

General rule:

• Process context → mutex or spinlock

• Interrupt/atomic context → spinlock only

Mental Model

Think of the locks this way:

Mutex

"I can't get the lock. I'll sleep until someone wakes me."

Spinlock

"I can't get the lock. I'll keep checking until it becomes available."

That single difference determines almost every practical use case.

Key Takeaways

• Both mutexes and spinlocks protect critical sections.

• Mutex losers sleep and later wake up.

• Spinlock losers remain active and wait.

• Mutexes have higher overhead because of context switching.

• Spinlocks are ideal for short, non-blocking operations.

• Mutexes are ideal for longer operations that may sleep.

• Never use a mutex in interrupt or atomic context.

• Never sleep while holding a spinlock.

Choosing the correct lock is one of the most important design decisions in Linux kernel development. Understanding when threads sleep and when they spin is the foundation of writing safe kernel code.

While studying concurrent network programming, I explored how to implement a worker pool in F# using MailboxProcessor, F#'s built-in actor abstraction.

The example launches multiple workers that process port numbers concurrently. A CountdownEvent is used to wait until all work items have been completed before shutting down the workers gracefully.

This approach is conceptually similar to Go's goroutines and channels but follows a more actor-oriented design.

Core Components

The implementation consists of three main pieces:

Defining Messages

Workers communicate through strongly typed messages.

type Message =
    | Port of int
    | Stop

This provides a clear protocol:

• Port → process a port number

• Stop → terminate the worker

Unlike string-based messaging, the compiler verifies message correctness.

Creating a Worker

Each worker runs as an independent actor.

let worker (id: int) =
    MailboxProcessor.Start(fun inbox ->
        let rec loop () = async {
            let! msg = inbox.Receive()

            match msg with
            | Port p ->
                printfn "Worker %d processed port %d" id p
                return! loop ()

            | Stop ->
                ()
        }
        loop ()
    )

Important characteristics:

• Sequential message processing

• No explicit locking

• Independent execution context

• Message-driven behavior

Tracking Completion

To emulate Go's WaitGroup, the example uses CountdownEvent.

let completed = CountdownEvent(1024)

Each processed port decrements the counter.

completed.Signal() |> ignore

The main thread blocks until all tasks are finished.

completed.Wait()

Creating the Worker Pool

A pool of 100 workers is created.

let workers =
    [|
        for i in 1 .. 100 ->
            worker i
    |]

Each worker owns its mailbox and processes incoming messages independently.

Dispatching Work

Port numbers are distributed across workers.

for port in 1 .. 1024 do
    let index = port % 100
    workers.[index].Post(Port port)

This creates a simple round-robin scheduling strategy.

Graceful Shutdown

After all work has completed, a termination message is sent.

for w in workers do
    w.Post(Stop)

This avoids abruptly terminating active workers.

Go vs F# Concurrency

The design maps naturally to concepts familiar to Go developers.

Although the implementation style differs, both approaches rely on message passing rather than shared-state synchronization.

Why MailboxProcessor?

MailboxProcessor offers several advantages:

• Actor-style concurrency

• Strongly typed messages

• No manual lock management

• Clear separation of worker responsibilities

• Scales well for network and backend services

This makes it a useful building block for:

• Port scanners

• TCP servers

• Telemetry systems

• Chat applications

• Distributed services

• IoT backends

Conclusion

F#'s MailboxProcessor provides a concise and powerful way to implement concurrent worker pools. By combining actors with typed messages and synchronization primitives such as CountdownEvent, I can build systems that are both scalable and easier to reason about than traditional shared-memory approaches.

Concurrency is everywhere inside the Linux kernel. Multiple execution contexts may attempt to access the same data simultaneously, leading to race conditions and corrupted state.

In this article, I build a simple kernel module that launches two kernel threads and protects a shared global counter using a mutex. Along the way, I explore:

• Kernel threads

• Shared resources

• Critical sections

• Race conditions

• Mutex synchronization

• mutex_lock_interruptible()

• Thread lifecycle management

The Problem: Shared Data

Consider a global variable shared by multiple kernel threads:

static int shared_resource = 0;

Both threads increment this variable repeatedly.

shared_resource++;

At first glance this looks harmless, but the operation is actually composed of multiple CPU instructions:

1. Read value

2. Increment value

3. Write value back

If two threads perform these steps simultaneously, updates can be lost.

Understanding Race Conditions

Without synchronization:

Thread-1 reads 5
Thread-2 reads 5

Thread-1 writes 6
Thread-2 writes 6

Expected result:

5 → 6 → 7

Actual result:

5 → 6

One increment disappears.

This is known as a race condition.

Enter the Mutex

Linux provides mutexes to guarantee mutual exclusion.

The module defines a mutex as:

static DEFINE_MUTEX(my_shared_mutex);

A mutex ensures that only one thread can enter a critical section at a time.

Critical Section Protection

The shared counter update is wrapped by the mutex:

if (mutex_lock_interruptible(&my_shared_mutex))
    return -EINTR;

shared_resource++;

mutex_unlock(&my_shared_mutex);

This guarantees that concurrent updates occur safely.

Why Use mutex_lock_interruptible()

Linux provides two common mutex acquisition APIs:

Example:

mutex_lock_interruptible(&my_shared_mutex);

Unlike mutex_lock(), the interruptible version can abort waiting and return an error if a signal arrives.

Return Values

A successful lock acquisition returns:

0

If interrupted before obtaining the lock:

-EINTR

This allows the thread to exit gracefully rather than waiting forever.

Creating Kernel Threads

The module launches two kernel threads.

thread1 = kthread_run(
    my_kthread_func,
    "Thread-1",
    "kthread_one");

thread2 = kthread_run(
    my_kthread_func,
    "Thread-2",
    "kthread_two");

Each thread repeatedly:

• Acquires the mutex

• Updates the counter

• Releases the mutex

• Sleeps briefly

• Repeats

Why task_struct Matters

Thread handles are stored as:

static struct task_struct *thread1;
static struct task_struct *thread2;

Every Linux task is represented internally by a task_struct.

These handles allow the module to:

• Stop threads

• Manage thread lifecycle

• Track execution state

Graceful Thread Shutdown

The worker loop uses:

while (!kthread_should_stop())

When the module unloads:

kthread_stop(thread1);
kthread_stop(thread2);

The threads detect the stop request and exit cleanly.

This is the recommended Linux kernel thread shutdown pattern.

Sleeping Locks

Mutexes are sleeping locks.

This means they may block and schedule another task while waiting.

Because of this, mutexes are appropriate in:

• Kernel threads

• Process context

• System call paths

They are not suitable for:

• Interrupt handlers

• SoftIRQs

• Atomic contexts

Sleeping is forbidden in those environments.

Mutex Ownership Rules

A mutex has strict ownership semantics.

Correct:

mutex_lock(&lock);

/* critical section */

mutex_unlock(&lock);

The same thread that acquires the lock must release it.

Violating this rule can trigger kernel warnings and undefined behavior.

Expected Kernel Output

After loading the module:

Initializing Mutex Kthread Module

Thread-1: Secured lock. Shared Resource value = 1

Thread-2: Secured lock. Shared Resource value = 2

Thread-1: Secured lock. Shared Resource value = 3

The counter increases sequentially because only one thread owns the mutex at any given moment.

Key Takeaways

• Shared kernel data requires synchronization.

• Race conditions occur when multiple threads modify data concurrently.

• Mutexes provide mutual exclusion.

• mutex_lock_interruptible() allows signal-aware waiting.

• Mutexes are sleeping locks and should not be used in interrupt context.

• Kernel threads are managed using task_struct.

• kthread_stop() enables clean thread termination.

Understanding mutexes is one of the foundational steps toward mastering Linux kernel synchronization and building reliable concurrent kernel code.

GitHub Repository: 👉 linux_kernel_mutex Explore the complete source code, build files, and module implementation on GitHub.

Scanning one TCP port at a time is simple, but it is inefficient. Network operations spend most of their time waiting for remote hosts to respond. Instead of scanning ports sequentially, we can launch multiple connection attempts concurrently and significantly reduce the total scan time.

In .NET, the Task Parallel Library (TPL) provides a lightweight mechanism for running large numbers of asynchronous operations without creating hundreds or thousands of operating system threads.

Why Concurrent Scanning?

A sequential scanner performs the following workflow:

Scan Port 1
Wait
Scan Port 2
Wait
Scan Port 3
Wait
...

This approach wastes time because the CPU sits idle while waiting for network responses.

A concurrent scanner launches many connection attempts simultaneously:

Port 1  ─┐
Port 2  ─┼─> Running Together
Port 3  ─┤
...
Port N  ─┘

The result is a much faster scan.

The F# Scanner

let scanPort port =
    task {
        try
            use client = new TcpClient()

            do! client.ConnectAsync("scanme.nmap.org", port)

            printfn "%d open" port
        with
        | _ -> ()
    }

What Happens Here?

If the connection succeeds, the port is considered open.

Launching 1024 Concurrent Scans

[1 .. 1024]
|> List.map scanPort
|> Task.WhenAll
|> fun t -> t.Wait()

This compact pipeline performs three important operations.

1. Generate Port Numbers

[1 .. 1024]

Creates a list of ports from 1 through 1024.

2. Create Scan Tasks

List.map scanPort

Transforms:

[1;2;3;...;1024]

into:

[Task1;Task2;Task3;...;Task1024]

Each task represents one asynchronous connection attempt.

3. Wait for Completion

Task.WhenAll

Combines all scan tasks into a single master task.

fun t -> t.Wait()

Blocks until every scan operation finishes.

Execution Flow

Ports List
     |
     v
Create Scan Tasks
     |
     v
Task1 Task2 Task3 ... Task1024
     |
     v
Task.WhenAll
     |
     v
Single Master Task
     |
     v
Wait()

Why Tasks Instead of Threads?

Creating 1024 operating system threads would be expensive.

Tasks provide:

• Lower memory consumption
• Better scalability
• Efficient I/O scheduling
• Simpler concurrency management
• Integration with async network APIs

For network-heavy workloads such as port scanners, Tasks are typically the preferred solution in modern .NET applications.

Final Thoughts

Concurrent scanning demonstrates a fundamental principle of network programming: don't wait on one connection when you can wait on many simultaneously.

Using TcpClient.ConnectAsync() together with Task.WhenAll() allows F# developers to build scalable network tools with surprisingly little code while taking advantage of the .NET runtime's efficient asynchronous I/O infrastructure.

Synchronization bugs are among the most difficult issues to debug in kernel-space software. A simple character device driver can become unsafe when multiple processes access shared resources concurrently.

In this article, I build and analyze a Linux character device driver that demonstrates four important kernel concepts:

• Mutex synchronization

• Concurrency control

• Character device drivers

• Linux version compatibility

The driver creates a device node named:

/dev/mydevice

and returns a simple message to userspace when read.

Concept 1: Mutex Synchronization

A mutex ensures that only one execution context accesses a critical section at a time.

The driver declares a mutex using:

static DEFINE_MUTEX(dev_mutex);

Before accessing shared state, the driver acquires the lock:

if (mutex_lock_interruptible(&dev_mutex))
    return -ERESTARTSYS;

and releases it when finished:

mutex_unlock(&dev_mutex);

Why Use a Mutex?

Without synchronization, multiple processes could simultaneously execute the read handler and modify shared data.

Benefits of mutexes include:

• Preventing race conditions

• Protecting shared kernel data

• Simplifying synchronization logic

• Allowing sleeping operations safely

Unlike spinlocks, mutexes can safely surround operations such as:

copy_to_user(...)

which may sleep.

Concept 2: Concurrency in Device Drivers

Concurrency occurs when multiple execution contexts attempt to access the same resource at the same time.

Consider several terminals executing:

cat /dev/mydevice

simultaneously.

Without synchronization:

• Multiple threads may enter the read handler together

• Shared state can become inconsistent

• Race conditions become possible

With mutex protection:

• Access becomes serialized

• Shared state remains consistent

• Device behavior becomes predictable

Kernel developers must always assume that a driver may be accessed concurrently.

Concept 3: Character Device Drivers

A character device transfers data as a stream of bytes between user space and kernel space.

Common examples include:

/dev/null
/dev/random
/dev/tty

Our module registers a character device using:

major_num = register_chrdev(
    0,
    "mydevice",
    &fops
);

Passing zero requests a dynamically allocated major number.

File Operations Interface

The kernel communicates with the driver through a file operations table:

static const struct file_operations fops = {
    .open    = dev_open,
    .read    = dev_read,
    .release = dev_release,
};

This maps standard system calls to driver callbacks.

Safe User-Space Communication

Kernel memory cannot be accessed directly from user space.

Instead, Linux provides helper APIs such as:

copy_to_user(buf, msg_data, len);

This function:

• Validates user-space addresses

• Handles page faults safely

• Prevents invalid memory access

It is the standard mechanism for transferring data from kernel space to user space.

Read-Once Device Behavior

The driver implements a simple read-once mechanism.

The key logic is:

if (*ppos > 0)
    return 0;

After the first successful read:

• File position advances

• Subsequent reads return EOF

• Behavior mimics many virtual kernel files

This pattern is commonly used in educational and demonstration drivers.

Concept 4: Linux Version Compatibility

Kernel APIs evolve over time.

A driver that compiles on one kernel version may fail on another if APIs change.

To support multiple kernels, the module uses:

#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,4,0)

For newer kernels:

class_create("mydevice_class");

For older kernels:

class_create(
    THIS_MODULE,
    "mydevice_class"
);

This allows a single codebase to compile across multiple Linux releases.

Important Kernel APIs

What This Project Demonstrates

This small module covers several important kernel-development concepts:

• Character device registration

• Dynamic device creation

• User-space interaction

• Mutex-based synchronization

• Concurrency protection

• Read-once semantics

• Cross-version kernel compatibility

Although simple, these concepts appear repeatedly in production Linux drivers.

Conclusion

Character device drivers provide an excellent introduction to Linux kernel development. Even a small driver can expose important topics such as synchronization, concurrency, memory safety, and API compatibility.

By combining mutex protection with a clean character-device interface, this example demonstrates how Linux drivers safely interact with user-space applications while maintaining correctness under concurrent access.

Source Code

GitHub Repository: 👉 linux_kernel_con_c_1 Explore the complete source code, build files, and module implementation on GitHub.

Port scanning is one of the most fundamental techniques in networking and security. Before diving into high-performance concurrent scanners, it's useful to understand how a basic scanner works by checking one port at a time.

In this article, we'll build the foundation of a nonconcurrent TCP port scanner using F#.

What Is a Port Scanner?

A port scanner attempts to connect to ports on a target system and determines whether those ports are:

TCP ports range from:

1 - 65535

For demonstration purposes, we'll scan only:

1 - 1024

These are commonly known as the well-known ports.

Step 1: Generate Target Addresses

Before attempting any network connections, we need a way to generate:

hostname:port

combinations.

In F#, a simple loop can accomplish this.

open System

for i in 1 .. 1024 do
    let address = sprintf "scanme.nmap.org:%d" i
    printfn "%s" address

Example output:

scanme.nmap.org:1
scanme.nmap.org:2
scanme.nmap.org:3
...
scanme.nmap.org:1024

Understanding sprintf

The sprintf function formats strings similarly to C's printf.

let address = sprintf "scanme.nmap.org:%d" i

Here:

For port 80, the generated string becomes:

scanme.nmap.org:80

Step 2: Attempt a TCP Connection

A TCP scanner determines whether a port is open by trying to establish a connection.

Conceptually:

Connect to target port
    |
    +-- Success -> Open
    |
    +-- Error -> Closed/Filtered

In .NET, this is typically done using:

System.Net.Sockets.TcpClient

Minimal example:

let client = new System.Net.Sockets.TcpClient()

If the connection succeeds, the port is likely open.

Step 3: Always Close Connections

A successful connection consumes operating system resources.

Good network citizenship requires closing connections immediately after testing.

Conceptually:

client.Close()

Benefits:

• Releases socket resources
• Prevents connection leaks
• Reduces system overhead
• Mimics professional scanner behavior

Scanner Workflow

The complete nonconcurrent scanning algorithm is straightforward:

for each port
    generate address
    attempt TCP connection

    if success
        print OPEN
        close connection

    else
        continue

Why Is It Called Nonconcurrent?

Only one port is tested at a time.

Port 1  -> wait
Port 2  -> wait
Port 3  -> wait
...
Port 1024

Advantages:

• Easy to understand
• Easy to debug
• Minimal code complexity

Disadvantages:

• Slow
• Network latency accumulates
• Does not scale to large scans

Modern scanners solve this using:

• Threads
• Async I/O
• Tasks
• Event-driven networking

Educational Value

Even though professional scanners use concurrency, a nonconcurrent scanner teaches several important networking concepts:

• TCP connection establishment
• Socket programming
• Port states
• Error handling
• Resource management
• Network reconnaissance fundamentals

Understanding this sequential model makes it much easier to appreciate how high-performance scanners achieve their speed.

Key Takeaways

• TCP scanners determine port availability by attempting connections.
• A loop can generate target addresses for thousands of ports.
• sprintf provides convenient address formatting in F#.
• Successful connections should always be closed.
• Nonconcurrent scanners are simple but relatively slow.
• This approach forms the foundation for advanced concurrent port scanners.

A nonconcurrent scanner may not be the fastest tool in a security engineer's toolkit, but it provides an excellent introduction to how network discovery and TCP-based reconnaissance actually work under the hood.

One of the best ways to understand TCP networking is by building a simple port scanner.

A port scanner attempts to connect to remote TCP ports and determines whether they are available. While the implementation may be small, it exposes several important networking concepts:

• TCP connections
• Client-server communication
• TCP handshakes
• Open and closed ports
• Exception handling
• Socket programming
• F# pattern matching

This article explores how a simple F# program can test TCP port availability and introduces the exception-handling syntax commonly used in networking applications.

What Is a TCP Port?

A TCP port is a logical communication endpoint.

Servers listen on ports waiting for incoming connections.

Examples:

When a client wants to communicate with a service, it attempts to establish a TCP connection to the target port.

What Is Port Scanning?

Port scanning is the process of testing one or more ports to determine whether they are accepting connections.

A scanner typically reports:

Port scanners are widely used in:

• Network administration
• Security auditing
• Asset discovery
• Troubleshooting
• Cybersecurity assessments

Understanding TCP Connections

Before data can be exchanged, TCP establishes a connection between client and server.

The client sends a connection request.

The server responds.

Once both sides agree, communication begins.

If the connection succeeds, the port is considered open.

If the connection fails, the scanner can infer that the service is unavailable or inaccessible.

Connecting with TcpClient

The .NET networking library provides the TcpClient class.

A connection can be created using:

use client =
    new TcpClient("scanme.nmap.org", 80)

This statement attempts to connect to:

• Host: scanme.nmap.org
• Port: 80

If the connection succeeds, the object is created successfully.

A Minimal TCP Scanner

The entire scanner can fit into a few lines of code.

try
    use client =
        new TcpClient("scanme.nmap.org", 80)

    printfn "Connection successful"

with
| :? SocketException ->
    printfn "Connection failed"

The scanner attempts a TCP connection.

Success indicates an open port.

Failure generates a networking exception.

Why Error Handling Matters

Networks are unpredictable.

Many things can go wrong:

• Host offline
• Service unavailable
• Firewall blocking traffic
• DNS resolution failure
• Routing problems

Instead of crashing, the application should handle these failures gracefully.

That is where exception handling becomes important.

Understanding the try Block

The try keyword marks code that might fail.

Example:

try
    connect()

Meaning:

Attempt this operation.
If an error occurs,
handle it below.

Networking operations are common candidates for try blocks because they depend on external systems.

Understanding the with Keyword

The with keyword begins exception handling.

Example:

try
    connect()

with

Meaning:

If an exception occurs,
look for a matching handler.

The runtime compares the exception against each rule that follows.

Understanding the Pipe Operator

The pipe symbol introduces a pattern-matching case.

Example:

| pattern -> action

Each pipe represents a possible match.

Example:

with
| case1 -> ...
| case2 -> ...
| case3 -> ...

The first matching case executes.

Understanding the Type Test Operator

One of the most important operators in F# exception handling is:

:?

This performs a runtime type check.

Example:

:? SocketException

Meaning:

Is this exception a SocketException?

If yes, the match succeeds.

If no, F# continues searching for another handler.

Understanding SocketException

SocketException is a .NET exception used for networking errors.

Namespace:

System.Net.Sockets

Typical causes include:

Because networking failures are common, SocketException appears frequently in TCP applications.

Understanding the Arrow Operator

The arrow operator is:

->

It means:

If this pattern matches,
execute the code on the right.

Example:

| :? SocketException ->
    printfn "Connection failed"

Meaning:

If the exception is a SocketException,
print an error message.

Execution Flow

Let's walk through the scanner step by step.

Step 1

Attempt TCP connection.

new TcpClient(...)

Step 2

Connection succeeds.

Output:

Connection successful

or

Step 3

Connection fails.

A SocketException is thrown.

Step 4

The exception handler matches:

:? SocketException

Step 5

The failure message is displayed.

Connection failed

Why Port Scanners Use Exceptions

Port scanners depend heavily on connection outcomes.

Successful connection:

Port Open

Connection refused:

Port Closed

Timeout:

Port Filtered

Host unreachable:

Target Unreachable

Exceptions provide a simple way to classify these outcomes.

Real-World Applications

TCP scanning forms the foundation of:

• Vulnerability assessment
• Service discovery
• Asset inventory
• Penetration testing
• Network troubleshooting
• Security monitoring

Understanding how scanners work also helps administrators better understand firewall behavior and network exposure.

Key Takeaways

• TCP scanners determine port availability by attempting connections.
• TcpClient provides a simple way to establish TCP sessions.
• Successful connections indicate open ports.
• Failed connections generate exceptions.
• SocketException represents common networking failures.
• try identifies code that may fail.
• with starts exception handling.
• | introduces pattern-matching cases.
• :? performs runtime type testing.
• -> specifies the action to execute after a successful match.
• Concurrency is essential for high-performance scanning.

Conclusion

Building a TCP port scanner is an excellent introduction to networking and systems programming.

Even a small scanner demonstrates several foundational concepts, including TCP connections, port states, socket programming, exception handling, and pattern matching.

For F# developers, understanding how SocketException integrates with pattern matching provides a clean and expressive way to handle networking failures while building reliable network applications.