Skip to content

The XNU network stack: mbufs, TCP, and Skywalk

From the classic 4.4 BSD TCP/IP stack to Apple's modern Skywalk replacement — how packets traverse XNU's networking code, and why Apple is moving the data plane out of the BSD layer.

Published 5 min read
bsdnetworkingtcpskywalkmbufkernelxnu

XNU's network stack is one of the most BSD-feeling parts of the kernel — mbufs, socket, sockaddr_in, the protocol switch table, all straight from 4.4 BSD with minor evolutions. But Apple has been quietly replacing the data plane with a new stack called Skywalk for years. This article walks both, and the relationship between them.

The classic stack — mbufs at the bottom

The atomic unit of networking memory in XNU is the mbuf — memory buffer. An mbuf is a small (typically 256-byte) structure that can hold:

  • A short inline payload (the small case — protocol headers, ACK packets).
  • A pointer to an external cluster (the large case — frame payloads up to a few KB).

mbufs are chained together — each carries a next pointer — so a complete packet is typically a small chain (header mbuf → cluster mbuf → cluster mbuf), letting protocols prepend their own headers without copying user data.

apple-oss-distributions/xnubsd/sys/mbuf.hThe mbuf structure — headers, flags, cluster references.View on GitHub(line —) apple-oss-distributions/xnubsd/kern/uipc_mbuf.cmbuf allocation, free, chaining, packet-header manipulation.View on GitHub(line —)

When a network packet arrives at a NIC, the driver:

  1. Allocates an mbuf (often from a per-CPU cache of pre-allocated ones).
  2. DMAs the packet into the mbuf's cluster.
  3. Hands the mbuf chain to the protocol input function via a queue.

From there, the packet walks up the protocol switch table.

The protocol switch table

Each protocol family (PF_INET for IPv4, PF_INET6 for IPv6, PF_LOCAL for Unix sockets) registers its operations: pr_input for received packets, pr_output for sent packets, pr_ctlinput for ICMP-style control messages.

A received IP packet:

  1. Driver hands the mbuf to ether_input.
  2. ether_input strips the Ethernet header, dispatches to ip_input (based on EtherType).
  3. ip_input validates the IP header, checks routing, and either forwards or hands up to L4.
  4. tcp_input or udp_input is called, finds the matching socket, queues the data.
  5. The blocked read(2) thread on that socket wakes.

apple-oss-distributions/xnubsd/netinet/ip_input.cIPv4 input path — fragmentation, validation, routing decision.View on GitHub(line —) apple-oss-distributions/xnubsd/netinet/tcp_input.cTCP receive — segment reassembly, ACK generation, congestion control.View on GitHub(line —)

The send path mirrors: app calls write(2) on a socket → tcp_output segments the data → ip_output adds the IP header and routes → ether_output formats for the link layer → driver enqueues for DMA to the NIC.

This is the FreeBSD-derived stack. The structure has been remarkably stable since the 1990s; the algorithms (RACK, BBR for TCP congestion control) have evolved continuously.

What's the problem with the classic stack?

A few:

  • Lots of locking. Every protocol layer takes per-socket and per-protocol locks. Under heavy load, lock contention dominates.
  • mbuf allocation cost. Even with per-CPU caches, mbuf alloc/free shows up as a significant fraction of network CPU time.
  • Userspace ↔ kernel data copies. The socket layer copies bytes between mbuf clusters and userspace buffers. For high-throughput apps (10 Gbps+) this is expensive.
  • The BSD stack is shared with iOS, where battery is critical. The classic stack's wakeup patterns don't align well with iOS's aggressive idle states.

So Apple built Skywalk.

Skywalk — the modern data plane

Skywalk is Apple's replacement for the network data plane. It's been shipping incrementally since macOS 10.15, and now handles the bulk of UDP, increasingly TCP, and the new QUIC-and-friends paths.

Key differences from the classic stack:

  • Kernel-to-userspace data path uses shared-memory rings, not byte-by-byte copies. Apps that opt in (Network.framework users) get zero-copy I/O.
  • Per-channel queues, not per-socket queues. Each network "channel" (Apple's term) is its own first-class queue, scheduled by the network stack rather than waiting on socket buffers.
  • Better integration with QoS. Network operations carry voucher-derived QoS hints down through the stack, so a foreground-app's connection out-prioritizes a background daemon's at every queueing point.
  • Userspace network extensions. Apple's Network Extension framework (content filters, DNS proxies, VPN clients) plug into Skywalk via well-defined extension points, replacing the older kernel-NKE (Network Kernel Extension) model.

Skywalk itself is closed-source — its kernel-side is not in the open XNU drop. Apple has occasionally published WWDC sessions explaining its architecture; for deeper detail, third-party reverse engineering covers the wire-level interfaces.

The classic stack still exists in XNU for compatibility — any app using legacy BSD socket APIs hits the classic path. New apps that adopt Network.framework get Skywalk by default.

Network Extension framework — replacing kernel NKEs

Before Skywalk, NKEs (Network Kernel Extensions) were how third-party software hooked into the network stack — for content filtering (Little Snitch), VPN tunneling, DNS interception. Like other kexts, NKEs are being phased out.

The replacement is Network Extension (NE) framework. NE providers:

  • Run as userspace processes (similar to DriverKit dexts).
  • Hook into the network stack via Mach-IPC-style extension points.
  • Get crash isolation, sandboxing, automatic restart.

Modern Little Snitch, Wireguard for macOS, every business VPN client — they all ship NE-based extensions now, not NKEs.

apple-oss-distributions/xnubsd/net/kpi_protocol.cNetwork protocol KPIs — the supported interface for third-party protocols.View on GitHub(line —) apple-oss-distributions/xnubsd/net/necp.cNECP — Network Extension Control Policy. The kernel side of the NE framework's policy enforcement.View on GitHub(line —)

Observing the stack on a live system

  • netstat — connection table, routing table, per-protocol statistics.
  • tcpdump — packet capture, runs on the classic stack's BPF interface.
  • nettop — per-process network usage.
  • nstat — modern Apple network statistics, includes Skywalk-only metrics.
  • dtruss -p <pid> -t connect — trace a process's network syscalls.

For Skywalk specifics, much less is exposed — most observability requires private SPIs or system trace tools.

What surprises newcomers

  • The classic 4.4 BSD stack is still in there. Compatibility with POSIX socket APIs means it can't be removed.
  • Skywalk is mostly invisible. Apps using Network.framework get it transparently; apps using BSD sockets don't.
  • NKEs are functionally dead on modern macOS. Third-party network tooling has migrated to Network Extensions.
  • Apple ships its own TCP congestion control variants. RACK + Apple-specific heuristics in Skywalk; classic CUBIC in the BSD stack.

apple-oss-distributions/xnubsd/netinet/tcp_subr.cTCP utility functions — connection table management, port hashing, RTT estimation.View on GitHub(line —) apple-oss-distributions/xnubsd/sys/socket.hThe socket API — every network syscall goes through this surface.View on GitHub(line —)

And the BSD personality article — networking is the biggest single BSD subsystem in XNU.

Related

What happens when you launch an app on macOS
From double-click to first window: LaunchServices, launchd, posix_spawn, AMFI, dyld, the shared cache, sandbox profile installation, the runloop. Six subsystems in three seconds.
bsdlaunchddyldamfi
How fork(), exec(), and posix_spawn work on XNU
The Unix-est of Unix calls, implemented on a Mach kernel. Why fork is awkward on macOS, what exec actually replaces, and why posix_spawn is now the preferred way to start a process.
bsdprocessforkexec
How macOS boots: iBoot to launchd
The full chain from power-on to your login window. Boot ROM, iBoot, kernelcache, kernel_init, bsdinit_task, launchd — what each stage does and how control transfers.
bsdbootlaunchdiboot