DeepSeek投机解码框架DSpark深度拆解：半自回归设计与置信度调度如何突破推理瓶颈

DSpark is a speculative decoding framework recently open-sourced by DeepSeek. The core goal of the paper is to solve two main bottlenecks in current speculative decoding technology: draft quality degradation and low verification efficiency. In simple terms, although the current draft model can quickly generate a long string of candidate tokens, due to a lack of context dependency, the quality of the draft deteriorates further along the sequence, and the acceptance rate drops rapidly. At the same time, regardless of the draft quality, blindly submitting the entire long draft to the target model for verification wastes valuable computational resources on tokens that are likely to be rejected.

To address these two problems, DSpark proposes two complementary innovative mechanisms:

Semi-Autoregressive Generation: To solve the draft quality problem, DSpark designs a hybrid architecture. It retains a computationally intensive parallel backbone network to quickly generate the 'skeleton' of all draft tokens at once; on top of this, it adds a lightweight sequential module that injects local dependency information in order while generating the draft. This design not only retains the high speed of parallel generation but also significantly improves the quality of the latter part of the draft by introducing correlations between tokens, slowing down the decay of the acceptance rate.

Confidence-Scheduled Verification: To solve the verification efficiency problem, DSpark introduces a dynamic scheduling mechanism. It trains a confidence head to predict the probability of each draft token being accepted. Then, a hardware-aware scheduler combines these confidence scores and the system's current real-time load to dynamically decide how long a draft to verify for each request. When the system load is low, longer drafts can be verified to pursue higher single-user speed; when the system load is high, low-confidence draft suffixes are automatically trimmed to avoid wasting compute on 'garbage' tokens, thereby stabilizing the overall system throughput.

Let's start with the principles of speculative decoding, why the result of speculative decoding with a draft model is equivalent to the result of target model inference. Imagine you are chatting online with a learned but slow-typing expert (like an old professor). This old professor is our target model. Its characteristics are: authoritative but slow. To speed up the process, we give him a fast-reacting doctoral assistant. This assistant is the draft model. Its characteristics are: agile but unreliable. Speculative decoding is this 'professor + student' collaborative workflow: The professor writes the first word, the student speculatively guesses the following content, the professor marks the draft, accepts and corrects, and completes a round. The core idea is to exchange a large number of guesses from a cheap, fast draft model for a small number of efficient validations from the expensive, slow target model.

The underlying mathematical principle is rejection sampling. We have a target distribution p and a draft distribution q. By sampling from q and accepting with a certain probability, we can perfectly recover sampling from p. The acceptance probability at each step is min(1, p(x) / q(x)). The proof shows that the final distribution is exactly the target distribution p. For a token x, there are two paths: either the draft token is accepted ('great minds think alike'), or it is rejected and resampled from a residual distribution ('seek another wise counsel'). The sum of the probabilities of these two paths exactly equals p(x).

For example, in the DSpark code deepspec/eval/base_evaluator.py, there is a common verification function verify_draft_tokens. The algorithm process is:

1. Send the sequence of draft tokens into the target model in parallel to get the probability distribution at each position.
2. For each draft token position, calculate the acceptance probability min(1, p_target(x)/q_draft(x)), and decide whether to accept by comparing it with a random number.
3. Use cumprod to implement prefix truncation: once a position is rejected, all subsequent positions are rejected.
4. If a rejected token exists, sample a new token from the residual distribution; if all are accepted, sample a bonus token from the last position's distribution of the target model.
5. Concatenate the accepted draft tokens and the newly sampled token as the final output.

Leviathan et al. established a theoretical analysis framework, revealing that the performance of speculative decoding is completely dominated by three core parameters:

Acceptance Rate α: Measures 'how reliable the draft is'. Formally, α = 1 - d_TV(p||q), where d_TV is the Total Variation Distance. Intuitively, α equals the 'overlap area' of the two distributions.

Cost Ratio c: Measures 'how cheap the draft is'. c = t_draft / t_target, the ratio of the draft model's single-step inference time to the target model.

Draft Length γ: Measures 'how many guesses per round'. If γ is too small, the speculative gain is limited; if too large, the acceptance probability of later tokens decreases exponentially.

The exact formula for the end-to-end speedup ratio is: (1 - α^γ) / (1 - α) / (cγ + 1).

From the 2022 paper 'Fast Inference from Transformers via Speculative Decoding', the algorithm uses an independent autoregressive model with far fewer parameters than the target model to serially generate γ candidate tokens. Later, 'Accelerating Large Language Model Decoding with Speculative Sampling' used a draft model with a wide and shallow structure. BiLD abandoned the requirement for lossless probability, allowing minor quality loss and introducing the concept of dynamic draft length, which is the origin of DSpark's confidence scheduling. Self-Speculative Decoding uses the original model's 'layer skipping' method for drafting, and the complete original model for verification.

SpecInfer breaks the limitation of a single candidate sequence, simultaneously considering multiple speculative candidates and organizing them into a token tree. It uses a tree parallel decoding mechanism to verify all sequences in the tree in a single LLM decoding step. Its draft model does not train a new model from scratch but directly uses pre-trained small models from the same family and uses a Boosting fine-tuning strategy. For the tree-based structure, by modifying the mask matrix method, the model can verify multiple sequences at once.

Medusa does not use a separate small model but adds multiple Medusa heads after the last layer's embedding of the original model. It combines the predictions of each head to build a tree and verifies all candidates in a single forward pass using a tree attention mask. EAGLE's core idea is to perform autoregressive prediction at the feature level and eliminate sampling uncertainty by inputting the 'one-step-ahead' token. Its draft model consists of an embedding layer, LM head, and an autoregression head, achieving an acceleration ratio much higher than traditional methods.

EAGLE-2 introduces a dynamically adjustable draft tree, improving upon EAGLE-1's static tree structure. Its algorithm mainly includes two stages: Expand and Rerank. The Expand stage selects nodes with the highest value to expand, and the Rerank stage sorts all draft tokens and selects the top-m ones with the highest value. EAGLE-3 uses feature sequences from low, middle, and high layers for fusion processing. By breaking the information bottleneck of single-layer features, it improves the model's ability to predict distant future tokens. It also proposes a Training-Time Test improvement to handle different inputs during training and testing.

Meta's MTP paper asks a question: why not imbue the model with speculative ability during the pre-training stage? By adding n independent output heads to simultaneously predict future n tokens, multi-token prediction not only improves the model's quality on downstream tasks but also allows these heads to directly act as drafts for speculative decoding during inference, achieving a unification of training enhancement and inference acceleration. DeepSeek-V3 MTP makes a key improvement by changing the parallel independent heads to sequential causal chains, which is closer to EAGLE's feature autoregressive concept. It proves that integrating multi-step prediction ability into pre-training can improve model quality and obtain a 'free lunch' for inference acceleration.

DFlash's breakthrough is this: why must the draft model be autoregressive? It repositions it as a parallel diffusion adapter. Its input is one anchor token plus γ MASK placeholders. Using bidirectional attention within the block, the model outputs predictions for all γ positions in a single forward pass. The key metric is that latency is independent of γ. Its second key innovation is KV injection: extracting hidden states from the target model at evenly spaced layers and fusing them into a contextual feature, then concatenating them into the Key and Value at each layer of the draft model. This allows the draft model to directly 'read' the target model's information at every layer.

During inference, after the target model performs a standard Prefill, the draft model generates all draft tokens in one forward pass. The target model then verifies the entire draft block in parallel. Unlike standard Speculative Sampling, DFlash adopts a simpler 'sample-then-compare' strategy because the diffusion model cannot produce per-position conditional probabilities. During training, it uses random anchor point sampling, sparse attention masks, and position-weighted loss to prioritize the optimization of earlier positions. Parameter sharing (embedding layer and LM head) is used to minimize trainable parameters.

DDTree does not discard the complete logits from the diffusion model's single forward pass. Instead, it builds a draft tree from these logits, selecting multiple paths under a fixed node budget B that are most likely to be accepted by the target model. Its core is the Best-First Heap search algorithm. It proves that a greedy algorithm is optimal: under the factorized distribution, the prefix probability is the product of probabilities at each position. By always expanding siblings and child nodes with the highest accumulated log-probability, it automatically satisfies the prefix closure property. The tree is then compiled into a format consumable by the target model (tree attention mask), allowing target model to verify the entire tree in one forward pass.

JetSpec addresses the fundamental contradiction exposed by DFlash: parallel generation sacrifices causality, causing the acceptance rate to decay rapidly along the draft block. JetSpec designs a tree-causal attention mask that achieves both parallel prediction and causal conditioning in a single forward pass. Each tree node can attend to itself and all its ancestors, but not to other nodes. This solves the 'causality-efficiency dilemma'. It uses a 5-layer draft head, feature fusion, and best-first search for tree building. It achieves a higher speedup ratio than DFlash+DDTree.

DSpark adopts a semi-autoregressive structure, dividing draft generation into two phases. The first is the parallel stage: a parallel backbone network runs a single forward pass over the entire block, producing hidden states and base logits. DSpark makes a minor modification: treating the anchor itself as the first prediction position, so γ input tokens produce γ draft logits. The second is the sequential stage: it supplements the base logits with a prefix-dependent transfer bias, allowing each draft position to condition on previously sampled tokens within the block. This is implemented via a Markov head (first-order dependency) or an RNN head (longer history), ensuring the sequential module is lightweight so total draft latency is still dominated by the parallel stage.

DSpark introduces a confidence head that outputs a scalar estimate for each draft position: the conditional probability that the draft token will pass the target model's verification, given all previous tokens in the block are accepted. This is supervised by the analytical acceptance rate. A post-hoc calibration technique called Sequential Temperature Scaling (STS) corrects the neural network's overly confident predictions to provide accurate absolute probabilities. The hardware-aware prefix scheduler formalizes the verification length selection problem as a global throughput maximization problem. It globally sorts all candidate tokens by their calibrated survival probability and greedily admits tokens until the system throughput no longer increases, dynamically finding the optimal operating point for the current system load.

Directly deploying Algorithm 1 into production exposes two fundamental conflicts: the discrete, stepped nature of real hardware capacity curves, and the need to know the batch size for the next step before the current step completes in zero-overhead scheduling. DSpark adjusts the scheduler to operate asynchronously. It uses the confidence head outputs from two steps ago to approximate the verification capacity for the next step. While approximate, this selection mechanism is order-preserving: the most confident draft tokens are always verified first. This design forms a causality barrier, maintaining lossless recovery of the target distribution while achieving maximum physical throughput. In production, DSpark replaces the MTP-1 baseline, achieving a 51-52% improvement in aggregate throughput under moderate SLA and significantly expanding the feasible interactivity frontier.

Experiments show that parallel drafting models (DFlash) and semi-autoregressive drafting models (DSpark) often achieve longer acceptance lengths than fully autoregressive ones. This stems from a capacity advantage at position 1: parallel models can afford deeper networks due to their γ-independent latency, resulting in significantly higher accuracy at the first position, which has the highest leverage in the prefix-matching survival process. Follow-up position independence limits parallel models, causing rapid acceptance rate decay. DSpark's semi-autoregressive design inherits the high initial acceptance rate of deep parallel models while its lightweight serial head mitigates the typical rapid decay, maintaining a high and stable conditional acceptance rate throughout the draft block.

Experiments under static threshold sweeps show that the confidence head can identify low-value suffix tokens, with pruning being most significant on chat workloads. In the 'Chat' subplot, increasing the threshold reduces rejected tokens significantly, raising the acceptance rate from 45.7% to 95.7%. Reliability diagrams show that while the original model achieves strong discrimination (ROC-AUC 0.81-0.90), it is overly confident (ECE 3%-8%). Applying post-hoc STS reduces the average ECE to about 1%, producing reliable survival estimates crucial for the hardware-aware scheduler.

The first principle of speculative decoding can be captured by a formula: the average wall-clock latency per generated token is (t_draft(γ) + t_target) / N(γ), where N is the expected number of accepted tokens. The speedup ratio is (1 - α^γ) / (1 - α) / (cγ + 1). Key properties: when α is close to 1, all candidates are accepted; when α is close to 0, it degrades to standard autoregression; N is extremely sensitive to α. There are only three levers: reduce draft latency, improve acceptance rate, and more intelligent verification. A fundamental trade-off exists between the first two: autoregressive drafting has linear latency growth but high α, while parallel drafting has latency independent of γ but lower α. DSpark's semi-autoregressive design approaches the latency of parallel while achieving α between the two.

The core question answered by the scheduling strategy is: given a batch of draft tokens, how many should be submitted to the target model for verification? This is formulated as an optimization problem: maximize the expected total accepted tokens multiplied by the hardware throughput at that batch size, by selecting the verification length for each request. DSpark's confidence scheduling uses the confidence head to estimate the conditional acceptance probability at each position. The prefix survival probability is the cumulative product of these per-step probabilities. The scheduler globally sorts all candidate tokens by this survival probability and greedily admits them until system throughput no longer increases, i.e., marginal benefit equals marginal cost.