Day 40: Task-Aware Routing — Assigning Models Based on Complexity

The Abstraction Trap

A junior engineer handed this problem reaches for LangChain’s RouterChain, LlamaIndex’s RouterQueryEngine, or some YAML-configured API gateway with a “smart routing” plugin. They wire it up, it ships to staging, and at 500 RPS everything looks fine.

At 50,000 RPS, the on-call pager goes off at 3 AM.

Here is what is happening underneath that elegant abstraction: every request materializes a Python object graph, calls a classification LLM (yes, another LLM to route to your LLM), waits for a JSON response, parses it, and then dispatches. You have added 40–120ms of P50 latency to every single request — regardless of whether the task is a three-word lookup or a 10,000-token summarization job. The framework made the simple case look clean and destroyed the hard case silently.

Worse: you cannot observe the failure because the framework ate it. The retry logic lives inside a black box. The routing table is Python heap memory that the GC owns, not you. There is no BPF map. There is no kernel-level telemetry. When the routing LLM starts hallucinating tier names, you find out from a customer — not from a metric.

We are going to build routing that makes classification decisions in under 400 nanoseconds at the XDP layer, before a packet has even touched the kernel’s sk_buff allocation path.

---

The Failure Mode: TLB Shootdowns and Scheduler Thrashing

Before we build, we need to understand exactly why the naive process-per-tier approach collapses at scale.

Failure Mode 1 — TLB Shootdowns. You run three model tiers as separate OS processes: nano-model, small-model, large-model. A Rust proxy receives requests, classifies in user-space, and sendmsgs to the appropriate backend. Each process owns its own page table. When the router switches from dispatching to nano-model (4MB working set) to large-model (2GB working set), the CPU must flush its Translation Lookaside Buffer for the old process’s virtual address mappings and populate new ones. On a 48-core Xeon, a TLB shootdown requires an Inter-Processor Interrupt to every other core that may have cached those mappings. The IPI round-trip costs roughly 1μs including cross-core synchronization. At 100k RPS with a typical 80/15/5 tier split, you are generating approximately 300k TLB shootdown IPIs per second — stealing 300ms of CPU time every second from your request budget.

Failure Mode 2 — CFS Scheduler Thrashing. Linux CFS has no knowledge that your nano-model task at nice 0 is latency-critical while your large-model task at nice 0 is throughput-oriented. They share the same run queue. A single 200ms large-model inference holds a core while 50 nano requests that would have completed in 4ms each sit in the ready queue. This is Head-of-Line blocking at the scheduler level — invisible to application metrics and inactionable without either cgroups CPU pinning or kernel SCHED_DEADLINE scheduling, neither of which your framework configures.

Failure Mode 3 — WASM Instance Heap Pressure. If you run multiple WASM instances in one host process (the naive Wasmtime approach), each instance has its own linear memory region. The component model’s shared-nothing guarantee means no memory is shared. Switching between a nano-instance (64MB heap) and a large-instance (4GB heap) on the same thread creates working-set pressure that evicts L2/L3 cache lines belonging to the hot nano path. On a 80/15/5 request mix, the 5% large requests are actively poisoning the cache for the other 95%. Measured on an Intel Xeon Platinum 8480+: L3 miss rate increases from 3% to 19% when large and nano instances share a host thread without CPU affinity.

---

The NexusCore 2026 Architecture

We solve all three failure modes with a three-layer pipeline that never makes a routing decision in user-space on the hot path:

The three failure modes are resolved structurally:

• TLB shootdowns: eliminated because all three tier pools run inside one Rust host process, sharing one page table. WASI’s shared-nothing isolation is enforced by the component model’s linear memory bounds, not by OS process boundaries.

• CFS thrashing: eliminated because the eBPF classifier fires at XDP time and writes to a pinned map. The routing decision costs zero scheduler interactions — it happens before the kernel decides which process to wake.

• Cache eviction: mitigated by pinning nano-pool threads to cores 0–7 (L3 slice A) and large-pool threads to cores 8–9 (L3 slice B) via pthread_setaffinity_np. Nano’s 64MB working set fits entirely in L3 slice A; large requests never evict it.

---

Implementation Deep Dive

GitHub Link :

https://github.com/sysdr/nexus-core-devops-engineering-p/tree/main/lesson39/nexuscore-day39

eBPF CO-RE: Compile Once, Run Everywhere

We use CO-RE via vmlinux.h BTF type information. The classifier compiles against BTF-annotated kernel types, and libbpf rewrites struct field offsets at load time for whatever kernel is present at runtime. No more #ifdef KERNEL_5_15 guards.

The complexity scorer in the XDP program operates in two passes over the packet payload, both bounded to avoid BPF verifier rejections:

Pass 1 — Token Estimation. score = payload_len >> 2. At an average of 4 bytes per subword token (empirically measured across English prose, code, and JSON), this gives a fast O(1) token count estimate with no memory allocation.

Pass 2 — Keyword Bitmap Scan. We inspect the first 64 bytes of TCP payload for high-complexity trigger prefixes: sum (summarize), ref (refactor), gen (generate), ana (analyze). Each match bumps the score by 10–25 points. The scan is a sequence of byte comparisons against the packet’s DMA buffer — no strchr, no copying, no heap.

The final score (0–255) is written into a pinned BPF_MAP_TYPE_HASH keyed on a 12-byte connection tuple: {src_ip u32, dst_ip u32, src_port u16, dst_port u16}.

BPF Map Pinning and Hot Reload

Pinning is the production-critical detail. A pinned BPF map at /sys/fs/bpf/nexuscore/ survives program reload. When you push an updated classifier with a tuned complexity threshold, the routing_decisions map is not reset — in-flight connections keep their tier assignment, preserving connection affinity through a live code update. Compare this to restarting a userspace proxy: every in-flight request races against the restart window.

Map sizing: 1,000,000 entries at 20 bytes each (12B key + 8B value) consumes 20MB of locked kernel memory. On a 256GB inference host, this is noise. The hash map uses open addressing with per-entry spinlocks (BPF_F_LOCK) for atomic updates. Per-CPU counters (BPF_MAP_TYPE_PERCPU_ARRAY) aggregate tier hit counts without atomic operations — each CPU core writes to its own slot and the Go loader sums across CPUs every two seconds.

WASI 0.3 Component Interface and Zero-Copy Body Transfer

Each model tier is a WASI 0.3 component implementing the wasi:http/proxy world:

world nexuscore-model {
  import wasi:http/types@0.2.2;
  import wasi:io/streams@0.2.2;
  export wasi:http/incoming-handler@0.2.2;
}

The critical production detail is wasi:io/streams InputStream.splice. When the router calls splice(body_stream, u64::MAX) on a component’s incoming body stream, the Wasmtime host implementation chains this to a kernel splice(2) call that moves bytes from the TCP socket’s receive buffer directly into WASM linear memory — one copy, kernel to WASM heap, zero additional user-space copies. At 10k RPS with a 4KB average body, naive Vec<u8> read-then-write would add 40MB/s of unnecessary memory bandwidth.

Memory limits are enforced per-instance via Wasmtime’s StoreLimiter trait:

• Nano: 64MB ceiling. Trap on memory.grow beyond limit.

• Small: 256MB ceiling.

• Large: 4GB ceiling. This is why pool-2: two 4GB virtual address spaces per host.

These limits solve the “runaway prompt” attack vector where a malicious tenant submits a request engineered to trigger unbounded memory growth in the model component.

TC Tail Calls and the skb→mark Signal

The TC eBPF program runs after XDP_PASS hands the packet to the kernel network stack. It looks up the connection tuple in the pinned routing_decisions map and writes the tier value into skb->mark. This mark is a 32-bit field in the socket buffer that survives up to the accept(2) call in the router process. The router reads it via SO_MARK on the accepted socket — no BPF map lookup in user-space, no syscall overhead beyond what accept already costs.

Tail calls in the TC program replace the current BPF stack frame rather than pushing a new one. The BPF verifier enforces a maximum chain depth of 33; our chain is 3 levels (classify → policy → mark). Each tail-called program shares the same ctx pointer and map file descriptors, enabling zero-copy context passing between classifier stages.

---

Working Demo Link :

Production Readiness: Metrics That Matter

Metric Target P99 Alert Threshold Tool XDP classifier latency < 800ns > 2μs bpftool prog profile BPF map lookup (hash) < 150ns > 500ns perf stat -e cache-misses WASI component cold start < 2ms (nano) / 8ms (large) > 15ms custom histogram Tier misroute rate < 0.1% > 0.5% tier_counters BPF map TLB shootdown IPI rate < 50k/sec > 200k/sec perf stat -e tlb:tlb_flush BPF tail-call depth ≤ 3 = 33 bpftool prog show Pool semaphore wait time < 500μs > 5ms Tokio metrics WASM memory page faults < 100/sec per instance > 1000/sec /proc/<pid>/stat

The tier misroute rate is your most important correctness signal. A nano task routed to large wastes GPU VRAM and increases queuing delay for legitimate large requests. Instrument it by comparing the eBPF-assigned tier against the complexity score reported in the model component’s response headers (X-NX-Tier).

Monitor TLB flush rate with perf stat -e tlb:tlb_flush -a sleep 5. If it climbs past 200k/sec despite all tiers living in one process, your WASM instance pool is undersized — you are evicting and re-instantiating components, which re-triggers page table population for each new instance’s linear memory mapping.

---

Step-by-Step Setup

Prerequisites:

# Kernel 5.15+ with BTF enabled
uname -r
cat /boot/config-$(uname -r) | grep CONFIG_DEBUG_INFO_BTF
# Must return: CONFIG_DEBUG_INFO_BTF=y

# Toolchain
sudo apt-get install -y clang-16 llvm-16 libbpf-dev linux-headers-$(uname -r)
rustup target add wasm32-wasip2
cargo install wit-bindgen-cli wac-cli
go install github.com/cilium/ebpf/cmd/bpf2go@latest

Generate and run the workspace:

cd nexuscore-routing
./start.sh        # Compiles eBPF, loads XDP+TC, starts Rust router
./demo.sh         # Fires one request per tier, prints tier headers
./verify.sh       # Asserts BPF maps loaded, router healthy, WASMs valid
./stress.sh       # 2500 nano + 500 small + 100 large, prints P99
./cleanup.sh      # Detaches XDP/TC, removes pinned maps, kills processes

Verify eBPF attachment:

sudo bpftool net show dev lo
# xdp name nexuscore_classify id <N> mode generic

sudo bpftool map show name routing_decisions
# type hash  key 12B  value 8B  max_entries 1000000  memlock 20MB

sudo bpftool map dump name tier_counters
# View per-CPU hit counts for each tier (summed by loader)

# Live complexity score tuning without program reload:
sudo bpftool map update name score_thresholds key 0 0 0 0 \
  value 50 130 0 0 0 0 0 0
# nano_max=50, small_max=130 — takes effect on next packet

---

Homework: Production-Level Challenge

The current complexity scorer uses a static heuristic baked into the XDP program. Your task is to implement adaptive threshold recalibration entirely in kernel-space using bpf_timer.

Requirements:

1. Add a BPF_MAP_TYPE_RINGBUF named latency_ringbuf (16MB). The Rust router writes a latency_event struct {timestamp_ns u64, tenant_id u32, tier u8, latency_us u32} to this ring buffer upon each component completion.

2. Write a BPF timer callback (attached via bpf_timer_init + bpf_timer_set_callback) that fires every 10 seconds. It reads the last 10,000 events from the ring buffer, computes P95 latency per tier, and if nano tier P95 exceeds 50ms (meaning tasks are too complex for nano), decrements score_thresholds.nano_max by 5, clamped at 20.

3. The threshold update must use bpf_map_update_elem with BPF_EXIST — never BPF_NOEXIST, which would silently fail if the key already exists.

4. Prove correctness: inject 1000 requests labeled with score=45 (just below the nano threshold) but with an artificial latency of 80ms (simulated by sleeping in the WASI component). After 20 seconds, run:

sudo bpftool map dump name score_thresholds

1. The nano_max field must have auto-decremented from 60 toward 40.

This is structurally identical to the adaptive bin-packing feedback loop Netflix’s Titus team shipped in 2024 for container right-sizing — except you are doing it for inference tier routing in 2026, with WASI components instead of container sidecars, and without a single user-space daemon polling kernel state.

---

Next: Day 41 — Distributed KV Consensus for Model Config: Implementing a WASI-native Raft log using wasi:sockets and wasi:io/poll — zero Tokio, zero Tonic, one shared BPF_MAP_TYPE_QUEUE as the commit channel.