Tables are the biggest latency bottleneck in document parsing. A dense table can take thousands of tokens to represent in HTML, including markup for every row and cell. Every token adds latency, and with standard decoding, each new token takes another pass through the model.
Any processing latency delays the work that depends on the data downstream, from matching bank transactions against a ledger to checking procedure codes on a medical bill or verifying carton counts against a freight manifest.
Some tables are highly repetitive: a credit card statement might list the same merchant and amount dozens of times, with only the date changing. But many tables, like financial summaries, are dense with numbers that follow no pattern, and a general speedup solution has to handle those too.
Our first approach to improving latency was n-gram speculation, which drafts upcoming tokens from earlier output so the model can verify several at once. n-gram made table parsing around 1.5–2× faster, but we didn't see latency gains on non-repetitive tables. We switched to DFlash, which uses a trained drafter instead, to better handle all table types. With an off-the-shelf DFlash drafter, median Light Table latency fell from 3.35 seconds to 1.76 seconds on a single H100.
After fine-tuning, acceptance length rose from 6.73 to 12.48 tokens per verification pass giving us a 4x speedup.
What is speculative decoding?
LLMs generate text one token at a time. Each token requires a full forward pass through the model, and each pass waits on the one before it. This results in high latency when parsing long outputs like tables.
Speculative decoding can reduce this latency by guessing ahead. A drafter proposes several tokens, and the target model checks the sequence in one pass. If the first five draft tokens match what the target would choose and the sixth diverges, we keep the first five, use the target's token at position six, and discard the rest of the draft. This saves time when drafting and verifying the block costs less than generating those tokens one at a time.
Acceptance length measures how many tokens each verification pass advances the output. A longer accepted sequence reduces the number of target-model passes needed to finish. When guesses fail early, drafting and verification can cost more than standard decoding.
Approaches differ mainly in the drafter. n-gram drafts from a history of earlier output. EAGLE uses a small model that drafts one token at a time, and Multi-Token Prediction (MTP) adds prediction heads that typically look only a couple of tokens ahead. DFlash uses a small model that drafts a whole block of tokens in one pass.
Why n-gram speculation works best on repetitive tables
Table HTML repeats the same tags across rows, and some tables repeat cell values too. We started with n-gram speculation to see how much of the next row we could draft from earlier output without training a separate model. As the table model generated HTML, we indexed sequences of tokens it had already emitted in a trie. At each decoding step, we looked up the most recent tokens in that history. If the same sequence had appeared before, we took the tokens that followed it and proposed them as the next part of the output. Drafting was a lookup, with no extra model pass.
This worked well on tables with repeated values as well as repeated markup. For example, if a statement lists the same merchant and amount several times, an earlier row can supply much of the next one. But a table can have a perfectly regular grid and a different value in every cell. N-gram might correctly draft the opening <td> tag, then copy an amount from an earlier row. Verification stops as soon as the amount diverges, so the drafter gets little benefit from the markup it guessed correctly later in the row.
In earlier production observations, n-gram averaged about five tokens per verification pass. We wanted a drafter that could make progress even when the next cell value hadn't appeared before.
How DFlash speeds up table parsing by drafting tokens in parallel
DFlash uses hidden states from several layers of the table model to condition a small drafter. We used a 16-position block. The first position is anchored by a token the target has already produced, and bidirectional attention lets the drafter predict the remaining positions in parallel. The table model then verifies the proposed tokens.
Since DFlash drafts from the model's representation of the input and output so far, it can propose a cell value that hasn't appeared in an earlier row.
| N-gram | DFlash | |
|---|---|---|
| How it drafts | Finds a matching sequence in earlier output and reuses what followed it. | Uses the table model's hidden states to predict a block of upcoming tokens. |
| Training requirements | No training needed. The lookup history builds as the model generates tokens. | A compatible pre-trained drafter. |
| Drafting costs | A lookup in the token history. | One forward pass through a small draft model per block. |
| What improves the draft | More useful matches in the token history. | Fine-tuning the drafter on outputs from the model that will verify its predictions. |
Off-the-shelf DFlash nearly halved median latency
Before training a DFlash head of our own, we tried one trained for the base model. We wanted to see whether it could still predict the next tokens after we had fine-tuned the target model to generate table HTML. We tested on a single H100 using a mix of real-world tables from business documents, averaging about 100 cells per table. With one request in flight, median latency fell from 3.35 to 1.76 seconds on Light Table and from 3.85 to 2.01 seconds on Performance Table, compared with speculation off. That's about 1.9× faster on each model.
This was very surprising result for us since we original believed that DFlash heads are model checkpoints specific and DFlash requires training to get any acceleration, especially since the input to DFlash are hidden states which gets influenced by training. Our hypothesis is that the fine-tuning made small alterations that DFlash head trained on base model can still transfer. This also means that base DFlash head can be transferred to other fine-tuned models as well and can be used without training. That result gave us a reason to train the head on HTML from our own table models and see how much further we could reduce latency.
Fine-tuning DFlash brought median latency down further
Training DFlash is different from regular training because the model is trained to predict what the main model will produce, not the ideal table output. To build its training data, we ran our training set through our fine-tuned model to capture what the model outputted. Then we taught DFlash to guess in the same way the main model would.
Fine-tuning took Light Table from 1.76 to 0.76 seconds and Performance Table from 2.01 to 0.96 seconds at the same concurrency. Compared with speculation off, that's a 4.4× speedup on Light Table and a 4.0× speedup on Performance Table. Average acceptance length rose from 5.13 to 12.12 tokens per verification pass on Light Table and from 6.73 to 12.48 on Performance Table.
At 16 concurrent requests, trained DFlash reduced median latency from 9.05 to 3.07 seconds on Performance Table and from 5.76 to 1.71 seconds on Light Table, compared with speculation off.
DFlash speeds up token generation after the first token
The model first processes the image and prompt in a phase called prefill. Then it decodes the HTML. Time to first token (TTFT) measures the wait before output begins. Time per output token (TPOT) measures how quickly generation progresses after that. On Light Table at one concurrent request, time to first token was 0.13 seconds both with speculation off and with trained DFlash, while time per output token dropped from 4.06 to 0.68 milliseconds, about 6× faster.
At 32 concurrent requests, trained DFlash still reduced time per output token from 15.54 to 5.00 milliseconds on Performance Table and from 9.38 to 3.70 milliseconds on Light Table. Time to first token rose from 0.22 to 0.40 seconds on Performance Table and from 0.20 to 0.30 seconds on Light Table. The decode gains still brought median request latency down on both models.
What's next
We started with tables because they were the biggest latency bottleneck in our parsing pipeline. Next, we want to see where speculative decoding can help with other structured outputs, like forms. Form fields have different patterns from table rows, so we'll measure how many tokens the drafter can get through each verification pass and whether that translates into faster requests.
We're also investigating further quantization to reduce table generation latency even more. We currently run at FP8, and moving to 4-bit could add more speed.
In the meantime, try table parsing on your documents today.
FAQ
Why are tables the biggest latency bottleneck in document parsing?
A dense table can take thousands of tokens to represent in HTML, including markup for every row and cell. Every token adds latency, and with standard decoding, each new token takes another pass through the model.
What is speculative decoding?
Speculative decoding reduces LLM latency by guessing ahead. A drafter proposes several tokens, and the target model checks the sequence in one pass. The target keeps the tokens that match what it would choose, supplies its own token where the draft diverges, and discards the rest. Acceptance length measures how many tokens each verification pass advances the output.
What happens when a drafted token is wrong?
The target model checks the drafted sequence in one pass. If the first five draft tokens match what the target would choose and the sixth diverges, it keeps the first five, uses its own token at position six, and discards the rest of the draft.
Why didn't n-gram speculation speed up every table?
n-gram speculation drafts from a history of earlier output, so it depends on tables repeating values as well as markup. It averaged about 5 tokens per verification pass and made table parsing around 1.5–2x faster. On a table with a different value in every cell, n-gram might correctly draft the opening tag, then copy an amount from an earlier row. Verification stops as soon as the amount diverges, so there was little latency gain on non-repetitive tables.
How is DFlash different from EAGLE and MTP?
EAGLE uses a small model that drafts one token at a time, and Multi-Token Prediction (MTP) adds prediction heads that typically look only a couple of tokens ahead. DFlash conditions a small drafter on hidden states from several layers of the target model and uses bidirectional attention to draft a whole block of tokens (16 positions, in Extend's setup) in one pass.
How much faster is DFlash without fine-tuning?
An off-the-shelf DFlash head trained for the base model made median latency about 1.9x faster on a single H100 with one request in flight: from 3.35 to 1.76 seconds on Light Table and from 3.85 to 2.01 seconds on Performance Table, compared with speculation off.
How do you train a DFlash drafter for a fine-tuned model?
DFlash is trained to predict what the main model will produce, not the ideal table output. Extend ran its training set through its fine-tuned model to capture what the model outputted, then taught DFlash to guess the same way. Average acceptance length rose from 5.13 to 12.12 tokens per verification pass on Light Table and from 6.73 to 12.48 on Performance Table, a 4.4x and 4.0x median latency speedup over speculation off.
Does DFlash still help at higher concurrency?
Yes. At 16 concurrent requests, trained DFlash reduced median latency from 9.05 to 3.07 seconds on Performance Table and from 5.76 to 1.71 seconds on Light Table. At 32 concurrent requests, it reduced time per output token from 15.54 to 5.00 milliseconds on Performance Table and from 9.38 to 3.70 milliseconds on Light Table. Time to first token rose from 0.22 to 0.40 seconds and from 0.20 to 0.30 seconds, but median request latency still came down on both models.
