Device Driver Developer

What a Device Driver Is Core

A device driver is kernel code that knows how to operate one kind of hardware and presents it to the rest of the system through a standard interface. A network driver makes a card look like a network interface, a block driver makes a controller look like a disk, and an input driver turns a touch controller into input events.

The driver has two faces. Downwards it programs registers, handles interrupts and sets up DMA. Upwards it implements callbacks that a kernel subsystem defines, such as ndo_start_xmit for networking or read and write for a character device.

Practice. Pick three devices in your computer and, using lspci, lsusb and lsmod, find which driver module handles each.

C for Kernel Code Core

Kernel C is ordinary C with house idioms you must recognise. There is no standard library: you use kernel versions such as kmalloc, memcpy and snprintf. Errors are negative errno values like -ENOMEM. Cleanup uses goto labels that release resources in reverse order.

Two macros appear everywhere. container_of(ptr, type, member) turns a pointer to an embedded member back into a pointer to the enclosing struct. ERR_PTR, IS_ERR and PTR_ERR encode an error number inside a pointer return. The C Roadmap covers the language itself.

Practice. Write container_of by hand with offsetof, then use it to walk from a list node to its parent struct in a user-space program.

Computer Architecture for Driver Writers Core

Drivers talk to hardware through memory-mapped I/O: device registers appear at physical addresses, and loads and stores to them are commands, not ordinary memory access. Devices signal the CPU with interrupts and move bulk data with DMA, reading and writing system memory directly.

You need a working model of caches, because the CPU and a DMA engine can see different copies of the same memory; of virtual memory, because the kernel maps physical addresses before using them; and of buses such as PCIe, USB, I2C and SPI, which differ in speed, discovery and addressing. The Computer Science Roadmap explains caches and virtual memory in depth.

Practice. Draw the path a single network packet takes from wire to application, naming the bus, DMA, interrupt and kernel layers it crosses.

User Space and Kernel Space Core

The CPU runs applications in an unprivileged mode and the kernel in a privileged one. Applications reach the kernel only through system calls, page faults and a few other entry points.

This split explains many driver rules. A pointer passed from an application is a user address that may be invalid, unmapped or malicious, so you never dereference it directly. A bug in a driver can crash the whole machine or open a security hole, because there is no protection between kernel components. Kernel stacks are small, usually a few pages, so large arrays on the stack are a mistake.

Practice. Use strace on cat reading a file in /sys and list every system call it makes before the data is printed.

Reading Datasheets and Register Maps Core

A driver is only as good as its author's reading of the hardware documentation. A register map lists each register's offset from the base address, its width, its reset value and the meaning of each bit field, with access types such as read-only, write-one-to-clear or self-clearing.

Read the whole section on initialisation, reset and interrupts before writing code, and look for the errata sheet, which lists hardware bugs you must work around. Define register offsets and fields with #define, and use the kernel helpers BIT(), GENMASK(), FIELD_GET() and FIELD_PREP() to extract and insert fields instead of hand-written shifts.

Practice. Take the datasheet of a common I2C sensor and write a header with every register offset and field defined using GENMASK.

Building and Booting the Kernel Core

You will build kernels constantly. The configuration lives in .config, edited with make menuconfig or started from a defconfig. Each option is built in, built as a module or left out. The build system is Kbuild: a driver adds a Kconfig entry and a line in a Makefile such as obj-$(CONFIG_FOO) += foo.o.

For Arm boards you cross-compile by setting ARCH and CROSS_COMPILE. QEMU lets you boot a fresh kernel in seconds with a small root file system, which is the fastest loop for experiments that do not need real hardware. The Linux Roadmap and Shell and Bash Roadmap cover the surrounding system and scripting.

Practice. Build a mainline kernel with a minimal config, boot it in QEMU with a busybox initramfs and print the version from inside.

Loadable Kernel Modules Core

A module is an object file the kernel links in at run time. It declares an init function, called on load, and an exit function, called on removal. You build it against a configured kernel tree and load it with insmod or, with dependencies resolved, modprobe.

static int __init hello_init(void)
{
	pr_info("hello loaded\n");
	return 0;
}
static void __exit hello_exit(void)
{
	pr_info("hello removed\n");
}
module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");

If init returns an error the module is not loaded, so init must undo anything it set up before failing. Modules must match the kernel they were built for, because the internal interface has no stability guarantee.

Practice. Build an out-of-tree hello module, load and remove it, and confirm the messages and the module's entry in /proc/modules.

Module Parameters, Licensing and Symbol Export Core

module_param exposes a variable that can be set at load time or on the kernel command line, and optionally through sysfs. Use parameters sparingly: per-device settings usually belong in device tree or sysfs instead.

MODULE_LICENSE matters in practice. Many core symbols are exported with EXPORT_SYMBOL_GPL and are available only to modules with a GPL-compatible licence, and loading a proprietary module marks the kernel as tainted, which affects how bug reports are treated. Your own module can export functions to others with EXPORT_SYMBOL or the GPL variant, which creates a load-order dependency that modprobe resolves.

Practice. Add a parameter to your module, change it through /sys/module, and check the taint flag after loading a module without a licence line.

Kernel Memory Allocation Core

kmalloc returns physically contiguous memory and takes GFP flags that say how the allocator may behave. GFP_KERNEL may sleep to reclaim memory, so it is allowed only in process context without spinlocks held. GFP_ATOMIC never sleeps and is used in interrupt handlers and under spinlocks, but it fails more often. kzalloc zeroes the memory, which avoids leaking old data.

vmalloc gives virtually contiguous memory for large buffers, but it is not usable for DMA and is slower to map. For many objects of the same size, a slab cache from kmem_cache_create is efficient.

Practice. Allocate with kmalloc inside a spinlock using GFP_KERNEL on a debug kernel and read the warning it produces, then fix it.

Kernel Data Structures Core

The kernel provides generic structures that you embed in your own structs. The doubly linked struct list_head is intrusive: the node lives inside your object and list_for_each_entry uses container_of to reach the object. Use the _safe variant when deleting during iteration.

Other common pieces: kref for reference counting with a release callback, the xarray and IDA for mapping integers to pointers or allocating minor numbers, red-black trees for ordered lookup, kfifo for ring buffers, hash tables from hashtable.h, and bitmaps for flags.

Practice. Keep a list of allocated objects in a module, print them through a debugfs file and free them all safely on exit.

Error Handling and Cleanup Core

Every kernel function that can fail returns a negative errno or an error pointer, and every caller must check. The usual structure is a sequence of setup steps with labels that undo them in reverse order.

buf = kzalloc(size, GFP_KERNEL);
if (buf == NULL)
	return -ENOMEM;
ret = register_thing(buf);
if (ret)
	goto err_free;
return 0;
err_free:
	kfree(buf);
	return ret;

Return the error you received rather than inventing a new one, so the real cause reaches the caller. Use specific codes: -EINVAL for bad arguments, -EBUSY for a resource in use, -ETIMEDOUT when hardware does not respond. -EPROBE_DEFER in a probe function means a dependency is not ready yet and the kernel will retry.

Practice. Write a module whose init has four steps, inject a failure at each one, and confirm with kmemleak that nothing leaks.

Kernel Logging, printk and Dynamic Debug Core

printk writes to the kernel log, read with dmesg. In drivers prefer dev_err, dev_warn, dev_info and dev_dbg, which prefix the device name. Log errors once with useful context, keep normal operation quiet, use rate-limited variants in hot paths, and use dev_err_probe in probe so deferrals do not clutter the log.

Dynamic debug makes dev_dbg calls almost free when off and lets you switch them on per file, function or line at run time through its debugfs control file, or at boot with a parameter. For early crashes, earlycon sends output to the serial port before the normal console starts. Prints change timing and can hide races, so use tracing for timing-sensitive bugs.

Practice. Convert your module to dev_ helpers, add a rate-limited error, and enable dynamic debug for just one function.

Character Devices and file_operations Core

A character device is a byte stream reached through a node in /dev. The driver reserves major and minor numbers with alloc_chrdev_region, adds a struct cdev with a struct file_operations table, and calls device_create so udev makes the node. For one or two nodes, misc_register with a struct miscdevice does all of this in one call, and reviewers will ask why you did not use it.

You implement open, release, read, write, unlocked_ioctl, mmap and poll as needed. Per-open state goes in file->private_data, and owner = THIS_MODULE stops the module being removed while a file is open. Many processes can call these functions at once. Before creating a custom node, check whether input, IIO, hwmon or the LED class already fits.

Practice. Write a misc character driver backed by a 4 KB buffer with read, write and llseek, and test it with cat, echo and dd.

Copying Data To and From User Space Core

User pointers must never be dereferenced directly. Use copy_from_user and copy_to_user, which check that the address is a user address, handle page faults and return the number of bytes they could not copy. A non-zero result should become -EFAULT. get_user and put_user handle single values.

These functions can sleep, because the user page may need to be brought in from disk, so they are not allowed while holding a spinlock or in interrupt context. Copy into a kernel buffer first, then take the lock. Validate every length and offset before use: a read handler that trusts the count argument is a classic kernel information leak. Also fetch user data only once; reading a value twice lets a malicious thread change it between the check and the use.

Practice. Write a read handler that honours the offset and count correctly, then attack it from user space with huge counts and bad pointers.

ioctl Design Core

An ioctl is a device-specific command sent through unlocked_ioctl. Command numbers are built with _IO, _IOR, _IOW and _IOWR, which encode a type character, a number, the direction and the argument size, so mistakes are easier to catch.

Good ioctl design is mostly about the argument structs, because they become a permanent interface. Use fixed-size types such as __u32 and __u64, avoid pointers and native long so 32-bit programs on 64-bit kernels work, pad explicitly so there are no hidden holes, and include a flags field that must be zero now so the interface can grow. Return -ENOTTY for unknown commands. Check permissions for privileged commands. Before adding an ioctl, ask whether sysfs, netlink or an existing subsystem interface would be better.

Practice. Add a get-status and a set-config ioctl to your driver with a versioned argument struct, and call them from a C test program.

mmap of Device Memory Recommended

Implementing mmap lets an application map device registers or a buffer straight into its address space, avoiding a copy per access.

For physical memory such as a register region, remap_pfn_range or io_remap_pfn_range builds the page table entries in one go; set the page protection to uncached or write-combining with pgprot_noncached or pgprot_writecombine as the hardware needs. For DMA buffers, dma_mmap_coherent does the right thing for the platform. For memory you want to fill lazily, provide a fault handler in vm_operations_struct. Always check the requested size and offset against what the device owns, or a user can map memory they should not see.

Practice. Map a kernel buffer into a test program with mmap, write a pattern from user space and read it back from the driver.

Blocking I/O, Wait Queues and poll Core

When no data is ready, a read should either sleep or, if the file was opened with O_NONBLOCK, return -EAGAIN. Sleeping uses a wait queue: the reader calls wait_event_interruptible(wq, condition) and the producer, often an interrupt handler, calls wake_up_interruptible after making the condition true. If the sleep returns non-zero a signal arrived, and the driver returns -ERESTARTSYS.

The poll operation lets select, poll and epoll work: call poll_wait for each wait queue and return a mask such as EPOLLIN when data is available. The condition must be checked under the same protection as the data, or a wakeup can be lost between the check and the sleep. completion objects are a simpler choice for one-off events such as waiting for a command to finish.

Practice. Make your ring buffer reader block when empty, wake it from a timer, and verify poll works from a small epoll program.

sysfs, debugfs and procfs Core

sysfs, mounted at /sys, shows the device model as directories and files. Drivers add attributes with DEVICE_ATTR_RO, DEVICE_ATTR_RW and attribute groups, writing show handlers with sysfs_emit. The rule is one text value per file, and each file is a stable interface that should be documented in the ABI directory. Register groups through the driver structure so they exist before udev is told about the device.

debugfs, under /sys/kernel/debug, is for developer-only data such as register dumps and fault injection switches; it has no stability rules and production systems may not mount it. procfs is for process information and old system files, and new drivers should not add to it. Interviewers like this distinction: stable user interface in sysfs, debugging in debugfs, procfs only for legacy.

Practice. Expose a threshold through sysfs and a register dump through debugfs, and change the threshold from a shell script.

Buses, Devices and Drivers Core

The driver model has three main objects. A bus type, such as PCI, USB, I2C or platform, knows how to enumerate devices and match them to drivers. A device represents one piece of hardware found on a bus. A driver declares which devices it supports, usually with an ID table.

When a device and a driver match, the bus calls the driver's probe with the device; when the device goes away or the driver is unloaded, it calls remove. Matching can happen in either order, which is why drivers must not assume their hardware exists at module load. MODULE_DEVICE_TABLE exports the ID table so udev can load the module when a matching device appears.

Practice. Browse /sys/bus/platform and /sys/bus/i2c on a board, and for one device find its driver, its modalias and its parent.

Platform Drivers and Probe Core

Devices on a system on chip are not discoverable: nothing on the bus announces them. The kernel represents them as platform devices, created from device tree or ACPI, and drivers register with platform_driver_register or the module_platform_driver macro.

A typical probe allocates a private struct, maps registers with devm_platform_ioremap_resource, gets the interrupt with platform_get_irq, obtains clocks, resets and regulators, initialises the hardware, requests the interrupt and finally registers with its subsystem. Registering last matters: as soon as the subsystem knows about the device, users can call into it. The remove callback reverses whatever probe did that is not managed automatically.

Practice. Write a platform driver that maps a register region and reads an ID register, bound through a device tree node in QEMU or on a board.

Device Tree and ACPI Core

Arm and RISC-V systems describe hardware the kernel cannot discover in a device tree: which devices exist, their registers, interrupts, clocks and GPIOs. The .dts source is compiled into a blob passed by the bootloader, and overlays add nodes at run time. Drivers match nodes through an of_device_id table of compatible strings. Each compatible needs a YAML binding in the kernel documentation, checked with make dt_binding_check and reviewed strictly, because it is a stable interface. Describe hardware, not driver settings.

x86 machines and Arm servers use ACPI tables instead, matched through an acpi_device_id table of hardware IDs. ACPI also defines power states and firmware methods. Use the unified property API, such as device_property_read_u32, so one driver works with both. Laptop device bugs are often in the firmware tables, which acpidump lets you read.

Practice. Write a device tree overlay for a sensor on a Raspberry Pi, load it, and read a custom property in probe with the unified property API.

Managed Resources with devm_ Core

The devm_ functions register a resource against a device so the driver core releases it automatically after remove or when probe fails. devm_kzalloc, devm_ioremap_resource, devm_clk_get, devm_request_irq and many others remove most error-path code from probe.

Two pitfalls come up in interviews. First, release happens in reverse order after remove returns, so an interrupt requested with devm_request_irq is still live while remove shuts down the hardware; the handler can run against a half-torn-down device. Disable the interrupt in remove or order things so this is safe. Second, memory from devm_kzalloc is tied to the device binding, not to open files, so a file still open after unbind can touch freed memory. Use devm_add_action_or_reset to put your own cleanup into the managed sequence.

Practice. Rewrite a probe function using devm_ helpers, then unbind the driver through sysfs while an interrupt is firing and check for problems.

MMIO with ioremap, readl and writel Core

Physical register addresses must be mapped before use with ioremap or, better, a managed helper. The result is an __iomem pointer, which the sparse checker refuses to let you dereference directly. Access goes through readl, writel and their 8, 16 and 64-bit variants.

These accessors do three things a plain pointer does not: they prevent the compiler from merging or reordering accesses, they handle the little-endian byte order most devices use, and they include barriers so that a write to a register is ordered after earlier writes to DMA memory. The _relaxed variants drop that ordering for speed in hot paths where you have reasoned it is safe. On PCI, writes are posted, so read a register back when you need a write to reach the device before continuing. regmap wraps register access with caching and locking for I2C, SPI and MMIO alike.

Practice. Write a helper that updates a register field with readl and writel under a spinlock, then convert it to regmap_update_bits.

Clocks, Resets, Regulators, GPIO and Pinctrl Core

On a system on chip, a block does nothing until it is powered, clocked, released from reset and its pins are routed to it. The kernel has a framework for each. The clock API (clk_prepare_enable), the reset controller API (reset_control_deassert), the regulator API (regulator_enable) and pinctrl, which usually applies the default pin state before probe, all take their configuration from device tree.

The GPIO descriptor API (gpiod_get, gpiod_set_value) handles reset and enable lines, including active-low logic described in firmware. A device that probes but never responds is very often missing a clock, a reset release or a power rail, so check these first during bring-up.

Practice. In your platform driver, get and enable a clock and an optional reset GPIO, and make probe defer cleanly if the clock is not ready.

Subsystem Frameworks: I2C, SPI, Input, IIO and More Core

Most drivers are clients of a subsystem rather than raw character devices. An I2C device driver uses i2c_driver and transfers with i2c_smbus_read_byte_data or i2c_transfer; an SPI driver uses spi_driver and spi_sync. Bus transfers sleep, so they are never done in hard interrupt context.

On top of the bus you pick the subsystem that matches the device's purpose: input for keys and touch, IIO for ADCs, accelerometers and other sensors, hwmon for temperature and fan monitoring, V4L2 for cameras, ALSA for audio, DRM for display and GPU, MTD for raw flash, and the LED, PWM and watchdog classes. Reviewers expect you to use the subsystem rather than invent your own interface.

Practice. Write an I2C driver for a temperature sensor that registers with hwmon, and read it with the sensors command.

Interrupt Handling and request_irq Core

A driver registers a handler with request_irq or devm_request_irq, giving the interrupt number, flags, a name for /proc/interrupts and a cookie passed back to the handler. The handler reads the device's status register, acknowledges the interrupt in the device, does the minimum urgent work and returns IRQ_HANDLED, or IRQ_NONE if its device did not raise it.

static irqreturn_t foo_irq(int irq, void *data)
{
	struct foo *f = data;
	u32 st = readl(f->base + FOO_STATUS);

	if ((st & FOO_IRQ_MASK) == 0)
		return IRQ_NONE;		/* shared line, not ours */
	writel(st, f->base + FOO_STATUS);	/* write one to clear */
	return IRQ_WAKE_THREAD;
}

Returning IRQ_NONE correctly matters on shared lines and lets the kernel detect a stuck interrupt. Know the difference between level-triggered and edge-triggered interrupts: forgetting to clear a level interrupt causes an interrupt storm, while edges can be missed if the handler is slow.

Practice. Request a GPIO interrupt on a board, count events in the handler and compare the count with /proc/interrupts.

Threaded IRQs Core

request_threaded_irq splits handling in two. The hard handler runs in interrupt context, checks and silences the device, and returns IRQ_WAKE_THREAD. The thread function then runs in a dedicated kernel thread where it may sleep, take mutexes and do I2C or SPI transfers.

This is the preferred design for most new drivers, and essential for devices behind slow buses, where even reading the status register sleeps. In that case pass a NULL hard handler with IRQF_ONESHOT, which keeps the line masked until the thread finishes so a level interrupt does not fire endlessly. The trade-off is a little extra latency from the context switch.

Practice. Convert your I2C sensor driver to use a data-ready interrupt with a threaded handler that reads the sample.

Bottom Halves: Softirqs, Tasklets and Workqueues Core

Softirqs are a fixed set of deferred handlers used by networking, block, timers and RCU; they run in atomic context and only core code defines them. Tasklets, built on softirqs, were the classic driver bottom half, but they cannot sleep and maintainers now ask for threaded IRQs or workqueues instead, so you mainly meet them in old code.

A workqueue runs a function later in a kernel worker thread, where it may sleep. Embed a struct work_struct in your private data, initialise it with INIT_WORK and call schedule_work or queue_work; delayed_work adds a timeout for polling and debouncing. Create your own queue with alloc_workqueue when you need ordering or high priority. Before freeing the struct that contains the work, call cancel_work_sync and make sure nothing can queue it again. Forgetting this is a very common use-after-free.

Practice. Use delayed work to poll a device every 100 ms, and confirm with KASAN that unloading the module mid-poll is safe.

Kernel Timers, Delays and Time Core

The kernel measures coarse time in jiffies, ticks of the system timer. Classic timers (timer_setup, mod_timer) run their callback in softirq context and suit timeouts that do not need precision. hrtimers give nanosecond resolution for things such as periodic sampling.

udelay and ndelay busy-wait and are for very short waits, especially in atomic context. usleep_range sleeps and suits waits of tens of microseconds to a few milliseconds, while msleep suits longer ones and may oversleep. For waiting on a register bit, use readl_poll_timeout or its atomic variant, which includes a timeout so a dead device cannot hang the kernel. Compare times with time_after to handle jiffies wrap-around.

Practice. Replace a hand-written busy loop that waits for a ready bit with readl_poll_timeout and test the timeout path.

Atomic Context Rules Core

Code is in atomic context when it must not sleep: in hard interrupt handlers, softirqs and tasklets, while holding a spinlock, with interrupts or preemption disabled, and inside an RCU read-side section. Sleeping there can deadlock the CPU or corrupt scheduler state, and the kernel reports "BUG: scheduling while atomic" or "sleeping function called from invalid context".

Things that may sleep include GFP_KERNEL allocations, mutexes, copy_to_user, msleep, I2C and SPI transfers, waiting for completions and most firmware loading. Functions that may sleep should call might_sleep() so debug kernels catch misuse even when the sleep does not happen. Enable CONFIG_DEBUG_ATOMIC_SLEEP during development. When you must do sleeping work in response to an interrupt, use a threaded handler or a workqueue.

Practice. Call msleep inside a spinlocked region on a debug kernel, read the full warning and identify each line of the stack trace.

Spinlocks Core

A spinlock is a busy-waiting lock for short critical sections, and the only lock usable in interrupt context. Holding one disables preemption on that CPU, so the code inside must not sleep.

The variant depends on who else takes the lock. If an interrupt handler takes it, process-context code must use spin_lock_irqsave and its restore pair, otherwise the interrupt can arrive on the same CPU while the lock is held and spin forever. If only softirqs share it, the _bh variants are enough. Inside the hard interrupt handler itself, plain spin_lock is fine. Keep critical sections to a few register accesses or list operations.

Practice. Share a counter between an interrupt handler and a read handler, protect it with spin_lock_irqsave and explain why plain spin_lock would deadlock.

Mutexes and Semaphores Core

A mutex is a sleeping lock: a waiter is put to sleep instead of spinning. Use it in process context for longer critical sections, especially those that sleep themselves, such as bus transfers, firmware loading or copying to user space. A mutex has one owner who must release it, and it cannot be used in interrupt context.

mutex_lock_interruptible lets a signal abort the wait, which suits file operations. Counting semaphores still exist but are rarely right in new code; a mutex or a completion usually expresses the intent better. Reader-writer locks (rwlock_t and rw_semaphore) help only when reads greatly outnumber writes and critical sections are long. A typical driver uses a mutex to serialise configuration and a spinlock for the small state shared with the interrupt handler.

Practice. Protect device configuration with a mutex and the interrupt status with a spinlock, and justify each choice in a code comment.

Atomic Operations and Reference Counting Core

atomic_t and its operations (atomic_inc, atomic_dec_and_test, atomic_cmpxchg) update a single integer without a lock. Bit operations such as set_bit, test_and_set_bit and clear_bit do the same for flags. They are fast and safe in any context, but they protect only that one value: two atomics updated together are not atomic as a pair.

Object lifetime uses reference counts. refcount_t and kref saturate instead of wrapping, which turns a counting bug into a warning instead of a use-after-free. Every path that keeps a pointer beyond a lock takes a reference, and the release function runs when the last one drops.

Practice. Give each open file a reference to the device state with kref, then unbind the device while a file is open and confirm nothing is freed early.

RCU Recommended

Read-copy-update lets readers access shared data with almost no cost while writers make changes by publishing new versions. Readers wrap access in rcu_read_lock and rcu_read_unlock and load pointers with rcu_dereference. A writer prepares a new copy, publishes it with rcu_assign_pointer, then waits with synchronize_rcu, or defers with call_rcu or kfree_rcu, before freeing the old one.

The key idea is the grace period: after every CPU has passed through a point where it could not be inside a read section, no reader can still hold the old pointer. RCU suits read-mostly data such as routing tables, lists of registered devices and configuration. Writers still need a lock among themselves. Readers must not sleep in classic RCU sections.

Practice. Protect a configuration struct read by an interrupt path with RCU, update it from an ioctl and free the old copy with kfree_rcu.

Memory Barriers and Ordering Core

Compilers and CPUs reorder memory accesses. x86 hides most of this, but Arm, RISC-V and PowerPC reorder freely, so a driver that works on a laptop can fail on a board. Barriers restore order where it matters.

For CPU-to-CPU ordering use smp_store_release and smp_load_acquire, or smp_wmb and smp_rmb in pairs, and READ_ONCE and WRITE_ONCE to stop the compiler tearing or caching a shared variable. For CPU-to-device ordering in DMA rings use dma_wmb after filling a descriptor and before setting its ownership bit, and dma_rmb after seeing the device's done bit and before reading the rest. Every barrier should pair with one on the other side, and a comment should say which.

Practice. Write a single-producer, single-consumer ring between an interrupt handler and a reader using acquire and release, and explain each ordering.

Lock Ordering, lockdep and Hangs Core

A deadlock needs a cycle: path A holds lock 1 and waits for lock 2 while path B holds 2 and waits for 1. Define a global order for nested locks and always follow it. Driver favourites: taking a lock without disabling interrupts when the handler also takes it, and calling cancel_work_sync while holding a lock the work item needs.

lockdep, enabled with CONFIG_PROVE_LOCKING, records every lock dependency and context it sees and reports a possible cycle the first time the order is broken, even if no hang happened, with stack traces for each link. For real hangs, the soft lockup, hard lockup and hung task detectors report stuck CPUs and tasks, and SysRq, through /proc/sysrq-trigger, dumps blocked tasks and held locks on a live system.

Practice. Create a two-lock inversion in a test module, read lockdep's report, then use SysRq to dump a task stuck on a completion.

DMA API: Coherent and Streaming Mappings Core

Devices see bus addresses, not CPU virtual addresses, so memory used for DMA must be mapped through the DMA API, which handles IOMMUs, bounce buffers and cache maintenance for you. First declare what the device can address with dma_set_mask_and_coherent.

Coherent memory from dma_alloc_coherent is visible to both CPU and device at all times, at the cost of being uncached on some platforms. Use it for long-lived structures such as descriptor rings. Streaming mappings with dma_map_single or dma_map_sg wrap an existing buffer for one transfer with a direction; ownership passes to the device until dma_unmap_* or dma_sync_*_for_cpu. Check results with dma_mapping_error. Never DMA to the stack or to vmalloc memory, and use scatter-gather lists for buffers that are not physically contiguous.

Practice. Allocate a coherent descriptor ring and map a streaming buffer in a test driver, and enable DMA API debugging to catch misuse.

Cache Coherency and DMA Pitfalls Core

On many Arm systems the device does not snoop CPU caches. If the CPU writes a buffer and the data is still in cache, the device reads stale memory; if the device writes memory while old lines are cached, the CPU reads stale data. The streaming API cleans or invalidates caches at map, unmap and sync points, which is why touching a buffer while the device owns it is a bug even if it seems to work on x86.

Other traps: a buffer that shares a cache line with other data can be corrupted when the line is invalidated, so DMA buffers should be cache-line aligned, which kmalloc generally guarantees for this purpose. Devices with 32-bit address limits may need bounce buffers through swiotlb, which costs performance. CONFIG_DMA_API_DEBUG catches missing unmaps, wrong directions and double mappings.

Practice. Explain in writing, step by step, what goes wrong if a driver reads a received buffer before calling dma_sync_single_for_cpu on a non-coherent system.

IOMMU, VFIO and User-Space Drivers Recommended

An IOMMU translates device addresses (IOVAs) through page tables, as the MMU does for the CPU. It isolates devices, because each can reach only memory mapped for it, and lets a device with a narrow address range use any memory. Intel VT-d, AMD-Vi and the Arm SMMU are common. With the DMA API it is mostly invisible; you notice it when a driver programs a physical address directly, or when IOMMU faults show a device touching memory after unmap.

The IOMMU makes VFIO safe: a user-space process or virtual machine gets direct access to a PCI device, including DMA. UIO is simpler, exposing registers by mmap and interrupts by read, but offers no DMA protection. User-space drivers suit fast development, isolation or kernel-bypass packet and storage processing.

Practice. Boot a VM with the IOMMU enabled, list /sys/kernel/iommu_groups and bind a spare PCI device to vfio-pci.

PCI and PCIe Drivers Core

PCI devices are discoverable: each has a configuration space with vendor and device IDs, class code, capabilities and up to six BARs that describe its register and memory regions. The kernel enumerates them at boot and on hot-plug.

A PCI driver provides a pci_device_id table and a probe that enables the device (pcim_enable_device), claims and maps BARs, enables bus mastering with pci_set_master so the device can do DMA, sets the DMA mask and allocates interrupt vectors with pci_alloc_irq_vectors. Prefer MSI-X or MSI, which deliver interrupts as memory writes and allow one vector per queue, over legacy shared INTx lines. Also handle error recovery callbacks for AER, and remember that a surprise removal makes register reads return all ones, a value drivers should treat as "device gone".

Practice. Write a PCI driver for the QEMU educational device: map its BAR, read its ID register, and trigger and handle an MSI.

USB Drivers Recommended

A USB device presents configurations made of interfaces, each with endpoints of four types: control for setup, bulk for large reliable transfers, interrupt for small periodic data and isochronous for streaming with bounded latency. Linux drivers bind to interfaces, not whole devices, through a usb_device_id table.

Data moves in URBs: allocate one with usb_alloc_urb, fill it for an endpoint, submit with usb_submit_urb and handle the result in a completion callback, which runs in atomic context. Simple synchronous helpers such as usb_control_msg and usb_bulk_msg may sleep. The hard part is disconnect: the device can disappear at any moment, so disconnect must kill outstanding URBs and stop new I/O while open files still exist.

Practice. Write a driver for a USB device you own that reads from an interrupt endpoint, then unplug it repeatedly during reads and fix any crash.

Network Drivers Recommended

A network driver allocates a net_device, fills net_device_ops with ndo_open, ndo_stop and ndo_start_xmit, and registers it. Packets travel as sk_buff structures. Transmit maps the buffer for DMA, writes a descriptor and rings a doorbell register; when the ring is full the driver calls netif_stop_queue and restarts it after completions free space.

Receive uses NAPI: the interrupt handler disables further receive interrupts and schedules polling, and the poll function processes up to a budget of packets, refills buffers and re-enables interrupts only when the queue is empty. This prevents interrupt storms at high packet rates. Modern adapters have many queues with receive side scaling, offloads for checksums and segmentation, statistics and ethtool operations.

Practice. Write a virtual network driver that loops packets between two interfaces using NAPI, and measure throughput with iperf.

Block Drivers and blk-mq Optional

A block driver exposes storage as a gendisk that the kernel reads and writes in sectors. Modern drivers use blk-mq, the multi-queue layer: you create a tag set describing hardware queues and depth, and implement queue_rq, which receives a request made of segments, maps them for DMA and submits them to the hardware. Completion calls blk_mq_end_request.

The block layer handles merging, scheduling and splitting, so the driver focuses on talking to the device quickly. NVMe is the reference design: one submission and completion queue pair per CPU with MSI-X per queue, which removes cross-CPU locking.

Practice. Read the null_blk driver in the kernel tree and trace a request from queue_rq to completion.

System Suspend and Resume Core

When the system suspends, the PM core calls each driver's suspend callbacks in an order that respects parent and child relationships, and resume callbacks in reverse. Drivers provide them through dev_pm_ops, usually with helper macros such as DEFINE_SIMPLE_DEV_PM_OPS.

Suspend must quiesce the device: stop new I/O, wait for in-flight work, save any register state the hardware will lose and disable interrupts or arm them as wake sources. Resume restores clocks and registers, re-initialises the device and restarts queues. Common bugs: a work item or timer still running across suspend, an interrupt firing before resume has restored state, and a device that loses its configuration because the driver assumed registers survive. Test with many suspend cycles in a loop, using the pm_test modes to isolate stages.

Practice. Add suspend and resume to a driver, then run 100 suspend and resume cycles with rtcwake and check the device still works.

Runtime Power Management Core

Runtime PM powers an individual device down while the rest of the system runs. The driver enables it with pm_runtime_enable and implements runtime_suspend and runtime_resume. Code that needs the hardware calls pm_runtime_resume_and_get before access and pm_runtime_put_autosuspend after, with an autosuspend delay so the device does not flap on and off.

Usage counts must balance on every path, including errors, or the device either never sleeps or sleeps while in use. Interrupt handlers cannot resume a device synchronously, so design for the device being active whenever its interrupt is enabled. Check /sys/devices/.../power/runtime_status to see what the device is doing.

Practice. Add runtime PM with autosuspend to your sensor driver and watch runtime_status change as you read from it.

ftrace and Tracepoints Core

ftrace is the kernel's built-in tracer, controlled through files in tracefs. The function tracer records every kernel function called; the function graph tracer shows call trees with durations, ideal for seeing why probe takes 300 ms. Event tracing uses predefined tracepoints in subsystems such as interrupts, scheduling, block and networking, with filters to keep the volume down.

trace_printk writes to the fast trace buffer instead of the console, so it barely disturbs timing, which makes it the right tool for races. Drivers can define their own tracepoints with TRACE_EVENT. trace-cmd and perf record and analyse traces, and interrupt-off and wakeup latency tracers find sources of latency.

Practice. Use the function graph tracer to time your driver's probe, then use irq tracepoints to measure how long its handler runs.

KASAN, KCSAN, UBSAN and kmemleak Core

Debug kernels catch whole classes of bugs automatically. KASAN detects out-of-bounds and use-after-free accesses and prints where the memory was allocated and freed. KCSAN detects data races between unsynchronised accesses. UBSAN catches undefined behaviour such as signed overflow and bad shifts. kmemleak scans for allocated memory that nothing points to any more.

These options slow the kernel and use more memory, so they belong in development and continuous integration, not production. Run every new driver on such a kernel with stress tests, unbind and rebind loops and fault injection, which can make allocations fail on purpose so you test error paths.

Practice. Introduce a use-after-free into a test module, catch it with KASAN and match each part of the report to your code.

Oops, Panics and Crash Dumps Core

An oops is the kernel's report of a fault in kernel code: the reason, such as a NULL pointer dereference, the faulting instruction as function plus offset, registers, the call trace and the list of loaded modules with taint flags. Translate offsets to source lines with scripts/faddr2line or scripts/decode_stacktrace.sh using a kernel built with debug information. The faulting address often explains the bug: a small address means a NULL struct pointer plus a field offset, and a pattern like 6b6b6b6b means freed slab memory.

A panic stops the machine. For crashes you cannot reproduce at your desk, kdump boots a second kernel to save memory to a dump file, which you examine with the crash utility: stacks of every task, kernel data structures and the log. KGDB allows source debugging over serial, and JTAG debuggers help during early bring-up.

Practice. Dereference a NULL private pointer in a module, then decode the oops to the exact source line with faddr2line.

Kernel Coding Style and Static Checks Core

The kernel has a written coding style: tabs eight characters wide, braces on the same line except for functions, short functions, lower-case names with underscores, no typedefs for structs and comments that explain why. scripts/checkpatch.pl checks most of it, and maintainers expect a clean result on new code.

Static analysis catches more. sparse, run with make C=1, flags mixing of user, kernel and I/O memory pointers and endianness mistakes through annotations such as __user, __iomem and __le32. smatch and Coccinelle scripts find common API misuse. Build with W=1 for extra warnings, and test that your code builds as a module and built in, on 32-bit and 64-bit, so the automated build robots do not find the problem first.

Practice. Run checkpatch with the strict option, sparse and a W=1 build on your driver and fix every warning.

Upstreaming Patches Recommended

Getting a driver into the mainline kernel means it is maintained with every release and other people fix API changes for you. The process is email based. Split work into a logical series, write commit messages that explain the problem and the solution, add Signed-off-by, and generate patches with git format-patch. scripts/get_maintainer.pl tells you which maintainers and lists to send them to, and git send-email sends them as plain text.

Expect review. Answer every comment, either with a change or a clear reason, and send a new version with a changelog below the --- line. Reply inline, not above the quoted text. New device tree bindings go to the binding maintainers as their own patch. The firm rule is that you must not break user space: an interface that programs rely on stays.

Practice. Prepare a two-patch series for a small fix in a driver you use, check it with checkpatch and get_maintainer, and send it to yourself first.

Windows Drivers: KMDF, UMDF and IRQL Optional

Windows drivers use the Windows Driver Frameworks: KMDF for kernel-mode drivers and UMDF for user-mode drivers, which run in a host process so a crash does not take down the system. Both wrap the older WDM model, which left Plug and Play and power handling to every driver. A KMDF driver creates a device object in EvtDriverDeviceAdd, much like probe, and I/O queues that deliver requests to callbacks. Drivers ship with an INF file, must be signed, and are tested with Driver Verifier and debugged with WinDbg.

Context rules use the interrupt request level. PASSIVE_LEVEL is thread context where code may wait. DISPATCH_LEVEL, where DPCs and spinlock holders run, forbids waiting and paged memory. Device interrupts run higher, at DIRQL. An ISR queues a DPC as a Linux hard handler wakes a bottom half, and touching paged memory at DISPATCH_LEVEL is the Windows version of sleeping in atomic context.

Practice. Make a table matching each Linux driver callback and context to its KMDF and IRQL equivalent.

Zephyr and RTOS Drivers Optional

Zephyr, an open-source RTOS for microcontrollers, borrows ideas from Linux. Hardware is described in device tree and features are configured with Kconfig, but everything is resolved at build time: the device tree becomes generated macros and devices are statically defined with DEVICE_DT_INST_DEFINE, with an init level and priority instead of probe ordering.

A Zephyr driver implements a subsystem API struct, for example for sensors, UART or GPIO, and applications call it through generic functions after checking device_is_ready. Interrupts are connected at build time, there is a single address space, and memory is usually static. Other RTOSes such as FreeRTOS have no driver model at all, so drivers are vendor libraries plus your own glue.

Practice. Build a Zephyr sample for a board you own and trace how a device tree node becomes a device pointer in the application.

Testing Drivers: KUnit, kselftest and Hardware Labs Recommended

Driver testing combines several layers. KUnit runs unit tests inside the kernel, often under user-mode Linux or QEMU, and suits pure logic such as parsers and calculations. kselftest holds user-space tests for kernel interfaces. QEMU emulates many devices, so you can test probe, interrupts and DMA paths without hardware, and emulated faults help test recovery.

Real hardware still matters. Teams run boards in a lab connected to power switches and serial consoles, so continuous integration can boot each kernel, run tests and collect logs. Stress tests should cover module load and unload loops, bind and unbind through sysfs, suspend cycles, hot-plug, and heavy I/O with fault injection.

Practice. Write a KUnit test for a helper that converts raw sensor values, and a shell script that binds and unbinds your driver 1,000 times.

Interview Preparation Strategy Core

Prepare in four areas. First, C fluency: write lists, ring buffers and bit operations quickly and correctly, and predict what snippets do. Second, driver concepts: be able to explain, with an example from your own code, probe and remove, atomic context, every lock type, deferred work, DMA mappings and barriers. Third, debugging: practise reading oops, lockdep and KASAN output from bugs you created on purpose. Fourth, design: for a described device, walk through bus and subsystem choice, user interface, interrupts, DMA, locking, power, error handling and testing, in that order.

Interviewers value honesty about the limits of your experience and a clear method more than a long list of APIs. Have two or three stories ready about hard bugs, with the symptom, the evidence, the cause and the fix. Use study plan to schedule revision and progress page to track it.

Practice. Record yourself explaining a driver you wrote, from device tree node to user-space interface, in under five minutes, then review the gaps.