How to evaluate document parsing APIs: a developer's framework

Every document parsing API looks accurate on a clean sample PDF. The differences show up on your hardest documents, at your real volume, inside your security requirements. This framework gives developers a repeatable way to evaluate document parsing APIs: build a test set, measure accuracy on the task the output feeds, check API design and deployment fit, and model the total cost at volume.

The short answer

Evaluate document parsing APIs on your own documents, against the task the output feeds, with every candidate run on the same frozen test set. The framework has seven steps:

1. Define the downstream task and what "correct" means for it.
2. Build a labeled test set of 50 to 100 of your own documents, including known failures.
3. Measure accuracy on messy documents with metrics that match the task.
4. Check API and SDK design for your integration pattern.
5. Check deployment, data handling, and compliance.
6. Model total cost at volume, including retries and human review.
7. Score every candidate on a weighted scorecard.

Extend is built to be evaluated this way: it publishes its own benchmark, RealDoc-Bench, offers evaluation sets through the API to score accuracy against your ground truth, and starts with 10,000 free credits and no sales call.

Key takeaways

• Accuracy only means something against a task. Parsing for retrieval, for field extraction, and for an agent each needs different metrics.
• Field-level accuracy hides document-level failures. A parser can get 97% of fields right and still get many documents wrong at least once.
• API design decides how much code you write around the parser: async processing, webhooks, batch limits, citations, and confidence scores.
• Price per page is not total cost. Retries, dropped pages, and review time often cost more than the parser.

The evaluation criteria at a glance

Step 1: define the downstream task

Write down what the parser's output feeds and what a correct result looks like:

• Retrieval and RAG: chunks must keep reading order, tables, and section structure so the right passage is found. Score retrieval hit rate or answer accuracy on questions about your documents.
• Field extraction: specific values must come out correct and traceable. Score field-level accuracy, and document-level accuracy for documents where one wrong value fails the whole record.
• Agents: the agent must complete its task from the output. Score task success and how often the agent needs a human.

This choice decides every metric that follows. A parser that wins on retrieval can lose on extraction.

Step 2: build a test set from your own documents

1. Collect 50 to 100 documents that cover every source, layout, and quality level you see in production. Coverage beats volume.
2. Include known failures on purpose: scans and phone photos, handwriting, multi-page tables, merged cells, rotated pages, long files, and mixed-language documents.
3. Label ground truth for the task: the expected values, the expected tables, or the questions and answers a reader should be able to answer from the output.
4. Tag each document by class (for example, "scanned bank statement" or "native PDF invoice") so you can score failures by class later.
5. Freeze the set. Keep it fixed so runs stay comparable as vendors ship new versions.

Step 3: measure accuracy on messy documents

Measure the parts of accuracy that break pipelines, separately:

• Field-level accuracy: each expected value scored on its own.
• Document-level accuracy: the share of documents with every required value correct. This is the number that decides how much review you need.
• Table structure: rows, columns, merged cells, and headers that carry across pages.
• Reading order and layout: multi-column text, footnotes, and form regions in the right order.
• Completion and dropped pages: files that fail, time out, or silently lose pages.

Use public benchmarks to build a shortlist, not to decide. Check that the data is public and resembles your documents. RealDoc-Bench scores parsers on real-world regulated documents from healthcare, financial services, logistics, and real estate. On it, Extend Performance Parse scored 95.7% field-level document Q&A accuracy across 581 documents and a 0.847 layout-adjusted F1 score across 1,500 layout samples, and the lower-cost Extend Light Parse scored 91.3%. Your own test set is still the deciding evidence.

Step 4: check API and SDK design

Build a thin integration with each finalist and check:

• Sync and async modes: a synchronous endpoint for interactive requests and an asynchronous one with webhooks for large files, so you don't poll.
• Batch processing: how many files a batch request accepts and whether batches slow down interactive traffic.
• Rate limits: requests per second and files per minute on the plan you'll actually buy.
• SDKs and API versioning: official SDKs in your languages and a pinned API version, so a vendor release doesn't change your output.
• Traceability: page numbers, bounding boxes, or citations for each value, so a reviewer can check a result without reopening the file.
• Confidence signals: per-field scores you can route on, sending low-confidence values to review.
• File coverage: the formats you receive, not just PDFs.

For reference, Extend accepts 35+ file types and 100+ languages, offers SDKs for Python, TypeScript, Java, and Go, supports async runs with webhooks and batch requests of up to 1,000 files, and allows 10 run requests per second on pay-as-you-go (rate limits). Extraction returns citations with the page and polygon for each value, plus confidence scores.

Step 5: check deployment, data handling, and compliance

Settle these before you spend time on accuracy, because any one of them can rule a vendor out:

• Where documents are processed: cloud regions, and whether the vendor can deploy into your own cloud.
• Data retention: how long files and outputs are kept, and whether you can set retention to zero.
• Certifications and agreements: SOC 2, HIPAA with a BAA, and GDPR, as your industry requires.
• Model training: whether your documents are used to train shared models.

Extend runs in US or EU cloud regions, can deploy into your own VPC, or can keep documents in your cloud while Extend runs inference. It supports SOC 2, HIPAA, and GDPR requirements and offers zero data retention. See Extend security for details.

Step 6: model total cost at volume

1. Convert every price to cost per 1,000 pages for the parsing mode you'd actually use.
2. Add failure costs: retries, failed files, and dropped pages you have to reprocess.
3. Add review costs: the share of documents a person must check, multiplied by the time per document. This is usually the largest line.
4. Check the commercial terms: free credits, minimums, contracts, and whether volume pricing needs a sales call.

Extend Parse costs $5 per 1,000 pages on Light and $20 per 1,000 pages on Performance on pay-as-you-go, with 10,000 free credits and no contract, monthly minimum, or sales call (pricing).

Step 7: score every candidate on a weighted scorecard

Weight each criterion for your use case, score every finalist from 1 to 5 on your test results, and multiply. Treat deployment and security as a pass-or-fail gate first if you're in a regulated industry.

How Extend supports this evaluation

Extend is built to be measured against your own documents:

• Evaluation sets in the API and Studio: attach files with expected outputs, run them against a processor version, and get overall and per-field accuracy, so a configuration change is verified before you publish it (evaluation overview).
• Automated iteration: Composer uses your evaluation sets to refine extraction configurations automatically.
• Published benchmarks: RealDoc-Bench reports parsing and Q&A accuracy on real-world regulated documents.
• Self-serve testing: sign up for an API key and 10,000 free credits, then run your test set the same day.

FAQ

How do you evaluate a document parsing API?

Define the downstream task the parser feeds, build a labeled test set of 50 to 100 of your own documents including known failures, and run every candidate on it with a frozen configuration. Score accuracy on what the task needs (field-level and document-level accuracy, table structure, reading order, dropped pages), then check API design, deployment and security fit, and total cost at your expected volume, including retries and human review.

What criteria matter most when comparing document parsing APIs?

Accuracy on your hardest documents comes first, measured on the task the output feeds rather than on clean samples. Then API design (async processing, webhooks, batch limits, SDKs, citations, and confidence scores), deployment and security (regions, private deployment, data retention, SOC 2, HIPAA, and GDPR), and total cost at volume. Weight them for your use case; a regulated team may treat deployment as a pass-or-fail gate.

How many documents do you need to evaluate a document parsing API?

Start with 50 to 100 documents that cover every layout, source, and quality level you see in production, including known failures such as scans, handwriting, multi-page tables, and long files. Coverage matters more than volume: 60 varied documents reveal more than 500 near-identical ones. Keep the set fixed so later runs stay comparable.

Should you trust published document parsing benchmarks?

Use them to build a shortlist, not to make the final call. Check whether the benchmark data is public, whether it resembles your documents, and what exactly it scores. Extend publishes RealDoc-Bench, where Extend Performance Parse scored 95.7% and Extend Light Parse 91.3% field-level document Q&A accuracy across 581 real-world documents, but your own test set is still the deciding evidence.

How do you compare document parsing API pricing?

Convert every price to cost per 1,000 pages at your expected volume and parsing mode, then add the cost of retries, failed pages, and human review. A cheaper parser that sends more documents to review often costs more in total. Extend Parse costs $5 per 1,000 pages on Light and $20 per 1,000 pages on Performance on pay-as-you-go, with 10,000 free credits and no minimum.

More guides

• 10 best document parsing APIs in 2026
• RealDoc-Bench report
• Best evaluation and benchmarking tools for document AI
• Document extraction API accuracy rankings