Skip to content

Kernel debugging on macOS: KDP, panics, and lldb-kdp

The path from a kernel panic to a stack trace — the panic handler, the KDP wire protocol, attaching lldb to a panicked kernel, and what's recoverable on a live system.

Published 5 min read
machkdpkerneldebuggingpanicxnu
XNU kernel panic flowPath of a kernel panic: trigger fires, state is saved, other CPUs halted, panic log written to disk, optional KDP wait for a debugger, then reboot.TRIGGERSunhandled trapnull deref, bad accessassert failurekernel invariant violateddouble faultfault while in fault handlerwatchdog timeoutscheduler stalledexplicit panic()from kernel codepanic() — osfmk/kern/debug.cevery panic routes through herepanic sequence1.save stateregisters, stack, panicreason, kexts, version →panic_info2.halt other CPUsIPI to every other core;only the panicking CPUruns3.write panic.log/Library/Logs/DiagnosticReports/Kernel_*.panic4.KDP wait or rebootif debug=0x100, poll forlldb; else rebootOUTCOMESno debugger → reboot, panic.log preserveduser sees the spinner + report on next loginlldb attaches via KDP → normal debuggingmemory, registers, breakpoints, single-step

When the XNU kernel panics, what happens? Where does the stack trace come from? Can you attach a debugger to a panicked kernel? What about a live one?

This article walks the kernel's debug machinery: the panic path, the KDP wire protocol, the relationship between lldb and a panicked Mac, and what's possible on a live system.

The panic path

A panic is XNU's way of saying "I cannot continue safely." Triggers include:

  • An unhandled CPU exception in kernel mode (null deref, bad address).
  • A failed assertion (assert(x) is a panic in production kernels too).
  • A double-fault (a fault while handling another fault).
  • A watchdog timeout (the kernel's heartbeat thread stops checking in).
  • An explicit panic("...") call from any kernel code.
apple-oss-distributions/xnuosfmk/kern/debug.cpanic() and the panic state machine — what every kernel-side abort routes through.View on GitHub(line —)

The sequence when panic fires:

  1. Save state. The CPU's register state, stack trace, panic reason, the kernel version, the loaded kexts are all captured into a panic_info structure.
  2. Stop other CPUs. Inter-processor interrupts halt every other core; only the panicking CPU runs from here on.
  3. Write panic.log to disk. The panic_info is serialized to /Library/Logs/DiagnosticReports/Kernel_<date>_<host>.panic (preserved across reboot).
  4. KDP wait (if enabled). The kernel waits for a debugger connection over Ethernet or USB-Ethernet for a few seconds. If a debugger attaches, normal lldb interaction happens.
  5. Reboot if no debugger and no debug boot-arg set.

KDP — the kernel debugger protocol

KDP is XNU's wire protocol for remote kernel debugging. Predates lldb (KDP was originally for gdb), now spoken by lldb's kdp plugin.

The protocol:

  • Runs over UDP at the wire level, typically over Ethernet or Apple's debug USB-Ethernet adapter.
  • Custom packet types: read registers, read memory, write memory, set breakpoint, continue, stop.
  • One-direction packets: the kernel is the target, lldb is the client. The kernel never initiates; it only responds.

apple-oss-distributions/xnuosfmk/kdp/kdp.cKDP packet handling — the kernel side of the debug wire protocol.View on GitHub(line —) apple-oss-distributions/xnuosfmk/kdp/kdp_protocol.hThe packet format definitions.View on GitHub(line —)

When the kernel is in a KDP wait:

  • All other CPUs are halted.
  • No timers fire, no IPIs are honored.
  • The KDP loop polls the network interface (in a tight loop, no interrupts) for incoming debugger packets.

This is why a panicked Mac with debug=0x100 boot-arg appears "hung" — the kernel is alive, sitting in the KDP wait, polling for a debugger that never connects.

Enabling KDP

By default, production kernels have KDP disabled. To enable for development:

sudo nvram boot-args="debug=0x141 kdp_match_name=en0"

The flag bits:

  • 0x001 (DB_HALT) — halt and wait for debugger on panic.
  • 0x100 (DB_KPRT) — print to a serial console during panic.
  • 0x040 (DB_ARP) — respond to ARP for the configured interface during KDP.

After reboot, the kernel will wait for a debugger on the named interface on panic.

To attach from another Mac on the same network:

lldb
(lldb) kdp-remote target-mac.local

lldb's kdp plugin handles the rest: packets fly, memory is readable, breakpoints can be set, kexts can be inspected.

Live kernel debugging

Live debugging (kernel still running, no panic) is much harder. KDP supports it but requires:

  1. Kernel built with debug symbols (Apple ships symbol packages separately).
  2. KDP enabled on the live kernel (the boot-args above).
  3. Sending the kernel an explicit "stop" via NMI (Non-Maskable Interrupt) — the kernel halts and enters KDP wait.

On modern Macs, an NMI can be triggered by:

  • A specific keyboard combo on the panicking Mac (depends on debug-build flags).
  • A debug USB-Ethernet adapter sending a magic packet.
  • A timer assertion in the kernel itself.

Once the kernel is stopped, lldb attaches just like on a panic and you can poke around.

Reading a panic log

When you don't have a debugger attached, the panic.log is what you get. The format:

panic(cpu 4 caller 0xfffffe002b40c8): "<message>" @<file>:<line>
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 24A123
Kernel version: Darwin Kernel Version 24.0.0: ...
Fileset Kernelcache UUID: ...

Backtrace (CPU 4):
  0xfffffe002b40c8: panic+0x84
  0xfffffe002b2014: trap+0x224
  ...

Process name corresponding to current thread: Foo
Boot args: -v debug=0x141

Mac OS version: ...

Last loaded kext at ...: <name>

The most useful parts:

  • Panic message — what the kernel was complaining about.
  • Backtrace — the kernel function call chain that led here. Each frame is a kernel address; with symbols, you can resolve to source.
  • Process name — what userspace process was current when the panic fired. Often the trigger.
  • Last loaded kext — third-party kexts loaded near the panic time. A frequent culprit.

To symbolize the addresses, you need a kernel symbols package matching the OS version. Apple ships these as Kernel Debug Kits (KDK) on developer.apple.com.

The watchdog

XNU has a watchdog thread that periodically checks "is the kernel making forward progress?" If it observes that the scheduler hasn't made progress in too long (a CPU is stuck spinning, deadlock, etc.), the watchdog panics on the assumption that the system is hung.

This is what catches "Mac frozen, mouse won't move, can't even SSH in." After a few seconds the watchdog times out and forces a panic + reboot, so the system at least comes back up rather than staying stuck forever.

apple-oss-distributions/xnuosfmk/kern/watchdog.cThe kernel watchdog — heartbeats and the panic-on-stall logic.View on GitHub(line —)

What surprises newcomers

  • A panicked kernel is alive but frozen — sitting in the KDP wait. The CPU isn't off; it's polling for a debugger.
  • Production kernels don't have full debug symbols built in. You need the KDK from Apple to symbolize panic logs.
  • KDP is unencrypted UDP. A KDP-enabled Mac on a hostile network is potentially compromisable. Only enable for development.
  • The watchdog is what reboots a frozen Mac. Without it, a hung system would just sit forever.

apple-oss-distributions/xnuosfmk/kdp/kdp_internal.hThe KDP internals — what state is exposed to lldb.View on GitHub(line —) apple-oss-distributions/xnuosfmk/kern/debug.hThe panic() function declarations and debug flag definitions.View on GitHub(line —)

For Apple's Kernel Debug Kit and the official kernel debugging guide, see developer.apple.com → Downloads → KDK.

And the boot article for context on how the debug boot-args change kernel behavior.

Related

Kernel synchronization in XNU: locks, atomics, and lock-free patterns
Spinlocks, mutexes, reader-writer locks, and lock-class groups — the synchronization primitives XNU offers, when each is appropriate, and how the per-CPU caches stay fast under contention.
machkernelconcurrencylocks
A context switch in XNU, walked end to end
From the moment an interrupt fires to the moment a different thread is running on the core — trap, AST, thread_invoke, ASID switch, return.
machschedulercontext-switchkernel
Mach IPC internals: kmsg, mqueue, vouchers
Inside a Mach message: how it's allocated, queued, woken on, and copied. Plus vouchers — the QoS-and-resource-propagation system most people don't notice.
machipckernelxnu