Draft, then verify
A lightweight proposer — a small model, a trained draft head, or the target’s own early layers — writes several candidate tokens ahead of the position being decoded. Drafting is cheap because the proposer is small.
The large target model then verifies the whole candidate block in a single forward pass, comparing each drafted token against what it would itself have produced. It accepts the longest valid prefix and discards the rest. One expensive pass advances several positions instead of one.
Every emitted token is checked by the target model, so the text is the text the target would have written alone: lossless under greedy decoding, distribution-preserving under sampling.[2][3] Speculation changes how fast the answer arrives, never what the answer is.
Long description
Two timelines. Above, sequential decoding: five successive target-model passes, each producing one token, each streaming the full weight set through memory. Below, draft-and-verify: a short run of cheap draft steps proposes five candidate tokens, then a single target-model verification pass checks them together and accepts the longest valid prefix. One expensive weight read advances several tokens instead of one. Mechanism per references 2 and 3.
One weight read, several tokens
An edge deployment decodes for one operator at a time. At batch size one the arithmetic units are idle most of the cycle: the constraint is memory bandwidth, because each token requires the model’s full weight set to be read from memory. The hardware is waiting on bytes, not on mathematics.
Verification changes the ratio. Checking a block of candidate tokens costs roughly one weight read, the same as producing a single token, so the expensive read is amortised across every token the target accepts. The governing variable is the mean accepted length — on Jetson AGX Thor, a Qwen3-8B target in NVFP4 with an EAGLE-3 draft head accepts 4.24 tokens per verification step, and throughput rises from 46.8 to 160.5 tokens per second, a 3.43× gain at batch size one.[1]
The same property sets the boundary. Under batching the arithmetic units are already busy, verification stops being free, and the technique inverts: the identical Thor configuration measured at 1.56× for one stream falls to 0.56× at batch eight.[1] Speculative decoding is a single-stream technique, and edge fusion is a single-stream problem.
Long description
Eight paired measurements on Jetson-class silicon, decode tokens per second at batch size one. Each row shows the vanilla baseline as a light bar and the speculative result beneath it. Qwen3-8B in NVFP4 with EAGLE-3 on Jetson AGX Thor rises from 46.8 to 160.5 tokens per second, a 3.43-times speed-up and the largest verified gain in the set. Qwen3-VL-4B reaches 3.25 times, Qwen3.5-27B with multi-token prediction 2.54 times, Qwen3-8B on AGX Orin 1.82 times, and on Orin NX 1.62 times. Two rows are regressions below one times: Qwen3.5-0.8B with multi-token prediction on Orin Nano falls from 55.3 to 45.0 tokens per second, and Qwen3.6-35B-A3B with DFlash on Thor falls from 87.6 to 42.2. All figures from the TensorRT Edge-LLM v0.9.0 benchmark publication.
Long description
Four families, ordered by the extra memory they require. Self-speculative decoding needs no extra weights: the target model drafts with its own early layers, is lossless, and yields modest gains (reference 9). Multi-token prediction adds a head trained with the network, costs little because it ships inside the checkpoint, and reaches about 97 per cent acceptance on the first two predicted tokens (reference 7). DFlash pairs a separate block drafter with the target at moderate cost, with a published starting depth of 15 for a block-16 checkpoint (reference 4). EAGLE-3 trains a draft head that reads the target’s own features, also at moderate cost, and holds the largest verified gains on Jetson silicon to date (references 1 and 6).
The deterministic path stays deterministic
Speculation requires something to speculate about. It applies wherever output is produced autoregressively — one element conditioned on those before it — because only then is there a next element a proposer can guess. That covers language generation, speech transcription,[10] time-series forecasting,[11] and action models.[12]
It does not accelerate deterministic signal processing. A fast Fourier transform, a filter bank, a beamformer: these compute a defined result from a defined input. There is no next token to propose and no verification to perform, because the answer is not being predicted. Applying speculation to that path would be a category error, and Enkidu does not attempt it.
The measurement path is deterministic and stays deterministic. Speculation lives only where a model was already predicting — it never touches the arithmetic that turns a sensor reading into a number.
A proposer and a verifier, by design
Baru — the trusted intelligence-fusion platform — runs its edge inference with draft-and-verify, because its regime is the technique’s best case: one operator, memory-bandwidth-bound hardware, long-form generation. The posture is set per tier rather than globally, since the measured envelope differs by an order of magnitude across the fabric.
The shape of the technique is also the shape of the platform’s ethos. A drafter proposes and is never trusted on its own; an authoritative model verifies every token before it is emitted; the decision that follows belongs to a person. Models propose; deterministic code or humans commit.
The Ekur capabilities surround the serving step rather than entering it. Anshar receives the acceptance rate as telemetry — an observability signal about scene stability before it is a tuning control. Nusku owns the configuration across the fleet, which matters because the same checkpoint has been observed at 0 per cent draft acceptance under one serving engine and 63 per cent under another.[8] Kittu’s trust checkpoints sit at the boundaries the request crosses, outside the loop entirely. Shedu is unaffected: speculation is interior to inference and pre-cryptographic.
Long description
Five tiers, rear to forward. The forward fusion centre on DGX Station GB300 is not in the Jetson benchmark set; long-form generation there is to be measured in the engineering programme. The local hub on Jetson Thor T5000 spans 1.06 to 3.43 times and is the primary speculation tier, with 70-billion-parameter-class models at 2.5 times. The heavy node on AGX Orin 64 GB spans 0.90 to 1.84 times at batch one, falling to 0.46 to 0.48 times at batch eight, so speculation is enabled on batch-one pipelines only. The standard node on Orin NX 16 GB spans 0.84 to 1.62 times and is selective per measured model class. The micro node on Orin Nano 8 GB spans 0.81 to 1.10 times and runs no local speculation, drafting upward to its hub instead.
Long description
A loop in four stages. The drafter proposes a block of candidate tokens. The target model verifies them and accepts a prefix. The measured acceptance rate — the mean accepted length per verification step — is emitted as Anshar telemetry, where it reads first as an observability signal about scene stability and only second as a control input. Nusku owns the resulting configuration across the fleet, because the same checkpoint can behave differently under different serving engines. Draft depth is then adjusted and the loop repeats. Kittu’s trust checkpoints sit outside this loop entirely, at the boundaries the request crosses.
The silicon these tiers run on is set out in the companion note, Silicon for the edge.
Speculative Decoding in Baru
Draft-and-verify inference in the trusted intelligence-fusion platform — the mechanism, the five places speculation lives in the platform, the Ekur touchpoints, the full verified benchmark set, and the measurement programme that follows.
In engineering review — not yet released for distribution
Request the whitepaperNo figure on this page is an Enkidu measurement. Every number is a published external benchmark on the silicon the platform’s tiers specify, stated with its version.