Benchmarks

AgentDocBench is an open, reproducible benchmark for "document → Markdown for agents". It measures what an agent needs from a converter: the words verbatim, in reading order, tables intact, fast, and in as few tokens as possible. It runs anymd, Docling, Marker, MarkItDown, Kreuzberg, Unstructured, and pdftotext on the same license-clean documents on GitHub-hosted runners.

anymd is one of the tools measured, and this benchmark lives in its repository. The scoring does not know which tool produced an output, the corpus and ground truth are public, and the categories where anymd loses are in the tables below like everything else. The harness, corpus, ground truth, and raw results are in bench/.

Choosing between converters? All comparisons puts the seven benchmarked tools side by side, and head-to-head pages compare anymd with Docling, Kreuzberg, Marker, MarkItDown, pdftotext and Unstructured on these documents, including where each one scores higher. We also publish architecture, privacy and pricing comparisons against LlamaParse, Mistral OCR and PyMuPDF4LLM.

Leaderboard

38 documents in 12 categories. Scores are 0–100; higher is better. Every failed, timed-out, or unsupported document scores 0.

Tool Overall Docs converted Sentences Text F1 Reading order Table cells F1 Time, all docs Output tokens
anymd 96.4 38/38 97.7 97.2 98.8 92.2 11.5 s 368,659
docling 93.0 38/38 94.4 94.5 94.4 89.9 2,432.4 s 430,148
kreuzberg 81.7 38/38 90.1 96.4 96.8 38.4 16.0 s 455,689
unstructured 81.2 38/38 92.0 97.1 93.9 38.4 346.5 s 414,333
markitdown 76.8 38/38 73.7 91.7 85.5 57.2 75.6 s 538,388
marker 71.0 30/38 78.9 79.9 76.8 60.9 7,104.5 s 277,779
pdftotext 42.2 23/38 60.6 71.4 52.0 0.0 0.90 s 201,448

Overall is the mean of the category scores below, so every category weighs the same. Time and tokens cover only the documents each tool converted; the table after the categories compares tools on the same documents.

By category

Category Docs anymd docling kreuzberg unstructured markitdown marker pdftotext
Papers (math) 3 96.7 91.5 82.5 80.3 74.2 56.7 83.3
Two-column papers 3 98.1 93.1 70.5 60.8 47.6 65.0 75.0
Reports 2 96.9 97.7 83.8 87.4 97.4 88.2 87.5
Financial tables 4 95.2 92.4 70.6 71.9 65.9 89.5 74.0
Forms 3 98.0 95.6 73.1 74.6 74.8 61.0 74.2
Scanned (OCR) 3 77.9 78.4 62.4 60.5 0.0 53.3 0.0
CJK 3 99.1 97.4 56.9 71.8 79.8 87.6 75.0
Slides 4 96.0 93.8 82.0 84.9 86.5 91.0 37.5
Spreadsheets 4 100.0 85.8 100.0 85.7 99.7 38.1 0.0
Word 3 99.9 99.9 99.8 99.8 99.8 94.7 0.0
EPUB 3 98.9 99.7 99.1 99.1 97.7 66.4 0.0
HTML 3 99.6 90.5 100.0 98.1 98.1 60.6 0.0

Where anymd loses: Reports (docling 97.7 vs 96.9); Scanned (OCR) (docling 78.4 vs 77.9); EPUB (docling 99.7 vs 98.9); HTML (kreuzberg 100.0 vs 99.6).

Speed and tokens on the 19 documents every tool converted

Tool Total time Median per document Output tokens
anymd 4.58 s 0.04 s 146,660
docling 1,723.3 s 63.9 s 162,290
kreuzberg 7.94 s 0.20 s 139,422
unstructured 187.4 s 7.74 s 145,360
markitdown 45.4 s 2.06 s 194,382
marker 5,256.3 s 161.7 s 181,132
pdftotext 0.66 s 0.02 s 147,183

Every document (score · time)

Document Category anymd docling kreuzberg unstructured markitdown marker pdftotext
paper-evsched paper 98.4 · 0.07 s 98.9 · 103.3 s 74.9 · 0.20 s 71.0 · 9.04 s 77.5 · 2.76 s 77.3 · 46.4 s 75.0 · 0.06 s
paper-rankone paper 94.8 · 0.06 s 90.7 · 353.6 s 98.0 · 0.41 s 95.9 · 11.7 s 75.6 · 3.51 s timeout 100.0 · 0.10 s
paper-ulam paper 96.9 · 0.02 s 84.8 · 65.0 s 74.7 · 0.20 s 74.0 · 10.1 s 69.4 · 2.06 s 92.8 · 600.6 s 75.0 · 0.04 s
twocol-bygpt5 two-column 96.7 · 0.10 s 93.6 · 74.7 s 62.2 · 0.39 s 63.5 · 7.74 s 47.5 · 4.03 s 99.0 · 73.2 s 75.0 · 0.12 s
twocol-smile two-column 99.1 · 0.08 s 91.6 · 70.4 s 74.7 · 0.31 s 58.2 · 9.88 s 45.1 · 3.13 s 95.9 · 112.2 s 75.0 · 0.06 s
twocol-uscore two-column 98.6 · 0.08 s 94.0 · 73.8 s 74.7 · 0.35 s 60.6 · 7.28 s 50.3 · 3.37 s timeout 75.0 · 0.10 s
report-crs-dod-approps report 94.8 · 0.04 s 96.1 · 42.4 s 74.5 · 0.26 s 74.9 · 8.40 s 95.8 · 2.86 s 77.5 · 297.3 s 75.0 · 0.03 s
report-plz report 99.0 · 0.01 s 99.3 · 22.9 s 93.0 · 0.10 s 99.9 · 5.34 s 99.1 · 1.63 s 98.9 · 10.8 s 100.0 · 0.01 s
fed-h8 financial 98.0 · 0.06 s 91.0 · 621.1 s 71.9 · 0.40 s 75.0 · 14.5 s 75.0 · 6.79 s 91.1 · 24.0 s 75.0 · 0.06 s
fin-bls-realer financial 88.3 · 0.02 s 87.7 · 32.0 s 65.9 · 0.15 s 68.3 · 5.80 s 55.6 · 1.69 s 84.1 · 428.0 s 75.0 · 0.02 s
fin-census-income financial 99.6 · 0.05 s 91.6 · 129.4 s 69.6 · 0.26 s 69.2 · 8.56 s 33.0 · 2.55 s 92.9 · 621.1 s 70.8 · 0.04 s
invoice-borderless financial 94.8 · 0.00 s 99.5 · 25.8 s 75.0 · 0.07 s 75.0 · 5.09 s 100.0 · 1.04 s 89.8 · 25.4 s 75.0 · 0.01 s
form-1040es form 97.5 · 0.05 s 94.6 · 101.8 s 69.8 · 0.22 s 74.1 · 10.2 s 74.5 · 5.26 s timeout 75.0 · 0.04 s
form-w4 form 97.7 · 0.03 s 92.5 · 63.9 s 74.7 · 0.16 s 75.0 · 6.35 s 75.0 · 3.34 s 94.4 · 594.8 s 72.7 · 0.02 s
irs-w9 form 98.9 · 0.03 s 99.8 · 37.2 s 74.8 · 0.15 s 74.7 · 7.84 s 74.9 · 2.71 s 88.6 · 508.7 s 75.0 · 0.02 s
scan-nasa-budget scanned 71.6 · 2.22 s 77.2 · 33.7 s 37.2 · 2.17 s 40.0 · 30.6 s 0.0 · 0.94 s 93.3 · 561.9 s 0.0 · 0.01 s
scan-nasa-notes scanned 62.1 · 1.69 s 58.1 · 32.6 s 66.7 · 2.11 s 58.1 · 25.6 s 0.0 · 0.95 s 66.7 · 524.6 s 0.0 · 0.01 s
scan-oklahoma-address scanned 100.0 · 6.59 s 100.0 · 56.5 s 83.3 · 5.52 s 83.3 · 64.1 s 0.0 · 0.97 s timeout 0.0 · 0.01 s
cjk-jp-stat-elderly cjk 99.1 · 0.03 s 98.0 · 70.8 s 52.8 · 0.14 s 66.7 · 7.23 s 97.2 · 1.73 s 97.6 · 10.4 s 75.0 · 0.02 s
cjk-tw-archives cjk 98.4 · 0.02 s 98.1 · 27.8 s 75.0 · 0.22 s 75.0 · 4.99 s 75.0 · 1.64 s 76.9 · 10.4 s 75.0 · 0.01 s
cjk-tw-dgbas-cpi cjk 99.9 · 0.04 s 96.0 · 50.9 s 42.9 · 0.43 s 73.7 · 7.73 s 67.1 · 1.86 s 88.3 · 11.7 s 75.0 · 0.03 s
slides-epa-1bp slides 91.7 · 0.01 s 92.1 · 6.00 s 86.9 · 0.07 s 90.0 · 3.47 s 91.5 · 1.05 s 92.7 · 86.7 s unsupported
slides-markitdown-pptx slides 98.5 · 0.00 s 99.0 · 8.54 s 93.9 · 0.06 s 100.0 · 4.76 s 97.1 · 0.98 s 91.0 · 396.8 s unsupported
slides-nasa-clarreo slides 94.5 · 0.04 s 95.2 · 89.6 s 74.5 · 0.13 s 74.4 · 5.62 s 82.4 · 2.12 s 91.8 · 161.7 s 75.0 · 0.08 s
slides-nist-pqc-update slides 99.3 · 0.04 s 89.0 · 129.8 s 72.7 · 0.09 s 75.0 · 6.94 s 75.0 · 1.55 s 88.6 · 633.1 s 75.0 · 0.02 s
csv-census-popest-2023 spreadsheet 100.0 · 0.00 s 100.0 · 5.99 s 100.0 · 0.06 s 100.0 · 1.05 s 100.0 · 0.95 s unsupported unsupported
csv-usgs-quakes-week spreadsheet 100.0 · 0.00 s 100.0 · 8.52 s 100.0 · 0.07 s 100.0 · 1.37 s 100.0 · 0.97 s unsupported unsupported
sheet-ers-crop-insurance spreadsheet 100.0 · 0.00 s 93.3 · 9.25 s 100.0 · 0.06 s 92.7 · 3.70 s 98.9 · 0.97 s 52.3 · 9.71 s unsupported
sheet-markitdown-xlsx spreadsheet 100.0 · 0.00 s 50.0 · 9.03 s 100.0 · 0.06 s 50.0 · 4.68 s 100.0 · 0.97 s 100.0 · 8.74 s unsupported
docx-hhs-market-research docx 100.0 · 0.01 s 100.0 · 6.40 s 100.0 · 0.06 s 100.0 · 3.43 s 100.0 · 1.50 s 87.9 · 735.9 s unsupported
docx-markitdown docx 100.0 · 0.00 s 100.0 · 8.16 s 99.5 · 0.06 s 100.0 · 4.69 s 99.5 · 1.07 s 97.4 · 73.8 s unsupported
docx-nsf-aaac-minutes docx 99.8 · 0.00 s 99.7 · 9.68 s 99.7 · 0.06 s 99.3 · 3.56 s 99.8 · 1.25 s 98.7 · 14.0 s unsupported
epub-gutenberg-jekyll epub 100.0 · 0.01 s 100.0 · 9.38 s 100.0 · 0.54 s 100.0 · 8.84 s 100.0 · 1.06 s 99.9 · 246.1 s unsupported
epub-gutenberg-relativity epub 100.0 · 0.01 s 99.9 · 6.89 s 100.0 · 0.08 s 100.0 · 6.50 s 100.0 · 1.23 s timeout unsupported
epub-markitdown epub 96.8 · 0.00 s 99.3 · 8.08 s 97.2 · 0.06 s 97.2 · 4.80 s 93.2 · 0.96 s 99.3 · 7.88 s unsupported
html-bls-ooh-software-developers html 98.7 · 0.00 s 93.7 · 9.44 s 100.0 · 0.07 s 94.4 · 4.84 s 94.4 · 1.10 s 83.2 · 182.9 s unsupported
html-markitdown-wikipedia html 100.0 · 0.01 s 88.9 · 10.6 s 100.0 · 0.10 s 100.0 · 6.22 s 100.0 · 1.37 s 98.5 · 85.5 s unsupported
html-wikipedia-roman-numerals html 100.0 · 0.02 s 88.9 · 7.47 s 100.0 · 0.14 s 100.0 · 3.90 s 100.0 · 1.66 s timeout unsupported

Tools

Tool Version Timed runs Run
anymd anymd 8.2.0 3 2026-10-01
docling 2.130.0 1 2026-09-25
kreuzberg 4.10.4 3 2026-09-25
unstructured 0.27.8 1 2026-09-25
markitdown 0.1.8 3 2026-09-25
marker 2.0.0 1 2026-09-25
pdftotext pdftotext version 24.02.0 3 2026-09-25

Machine: Linux x86_64, 4 CPUs (GitHub-hosted runner). pdftotext is the reference extractor for born-digital PDFs (bench/reference/), so its text F1 on those documents is 100 by construction; it outputs plain text, so it scores 0 on tables and on non-PDF formats.

How it is scored

Every output is normalized the same way (Unicode NFKC, Markdown and HTML markup removed, whitespace collapsed, curly quotes straightened, spaces between CJK characters dropped) and compared with the document's ground truth:

  • Sentences: key sentences copied verbatim from the document must appear verbatim. Glued words, split columns, or dropped lines fail the sentence. For scanned documents these are the sentences a correct OCR produces.
  • Text F1: bag-of-words F1 against a reference text (the PDF text layer, the DOCX/PPTX XML text, or the EPUB spine text). Recall drops when text is lost, precision when noise is added.
  • Reading order: headings and passage openings must appear in order. The score is the longest in-order chain divided by the number of strings, so two-column interleaving and scrambled sections both cost.
  • Table cells F1: every ground-truth table row is matched to the output row with the longest in-order run of equal cells. Recall is matched cells over truth cells; on complete tables, precision is matched cells over the cells of the output tables they came from. Pipe tables and HTML tables both count; plain text scores 0.
  • Document score is the mean of the scores that apply; category score is the mean over its documents, with a failed, timed-out, or unsupported document counting 0; overall is the mean of the category scores.
  • Time is the wall time of a fresh process per file (start-up included), median of 3 runs (1 run for tools that take minutes per document), after one untimed warm-up. Tokens are o200k_base.

The full method, the ground-truth schema, and the known limits are in bench/README.md.

Corpus

Every document is public domain, CC0, CC BY, CC BY-SA, MIT-licensed, or under an open government license, and is pinned by SHA-256. Generated from bench/corpus.json:

Document Category Format Pages License Source
paper-evsched paper pdf 23 CC-BY-4.0 Emily van Huffelen, Roel Brouwer, Marjan van den Akker (Utrecht University) (arXiv:2403.03109v1)
paper-rankone paper pdf 27 CC-BY-4.0 Khazhgali Kozhasov, Josué Tonelli-Cueto (arXiv:2201.02191v2)
paper-ulam paper pdf 11 CC-BY-4.0 Pat Devlin, Leo Douhovnikoff (Swarthmore College) (arXiv:2403.02276v1)
twocol-bygpt5 two-column pdf 16 CC-BY-4.0 Jonas Belouadi, Steffen Eger (arXiv:2212.10474v2)
twocol-smile two-column pdf 13 CC-BY-4.0 Miao Peng, Ben Liu, Qianqian Xie, Wenjie Xu, Hua Wang, Min Peng (arXiv:2210.04870v3)
twocol-uscore two-column pdf 17 CC-BY-4.0 Jonas Belouadi, Steffen Eger (arXiv:2202.10062v3)
report-crs-dod-approps report pdf 12 public-domain-us-gov Congressional Research Service (Cameron M. Keys)
report-plz report pdf 5 public-domain-us-gov U.S. Access Board
fed-h8 financial pdf 22 public-domain-us-gov Board of Governors of the Federal Reserve System
fin-bls-realer financial pdf 5 public-domain-us-gov U.S. Bureau of Labor Statistics
fin-census-income financial pdf 10 public-domain-us-gov U.S. Census Bureau (Gloria G. Guzman)
invoice-borderless financial pdf 2 MIT Microsoft Corporation (markitdown test files)
form-1040es form pdf 12 public-domain-us-gov Internal Revenue Service
form-w4 form pdf 4 public-domain-us-gov Internal Revenue Service
irs-w9 form pdf 6 public-domain-us-gov Internal Revenue Service
scan-nasa-budget scanned pdf 3 public-domain-us-gov Jesse L. Mitchell (Office of the Special Assistant for Science and Technology) / Dwight D. Eisenhower Presidential Library, NARA
scan-nasa-notes scanned pdf 3 public-domain-us-gov Robert Piland (White House science advisory staff) / Dwight D. Eisenhower Presidential Library, NARA
scan-oklahoma-address scanned pdf 7 public-domain-us-gov The White House (President Dwight D. Eisenhower) / Dwight D. Eisenhower Presidential Library, NARA
cjk-jp-stat-elderly cjk pdf 4 PDL-1.0 総務省統計局 (Statistics Bureau of Japan)
cjk-tw-archives cjk pdf 4 CC-BY-4.0 國家發展委員會檔案管理局 (National Archives Administration, National Development Council, Taiwan)
cjk-tw-dgbas-cpi cjk pdf 6 Taiwan-OGDL-1.0 行政院主計總處 (Directorate-General of Budget, Accounting and Statistics, Executive Yuan, Taiwan)
slides-epa-1bp slides pptx 27 public-domain-us-gov U.S. Environmental Protection Agency
slides-markitdown-pptx slides pptx 6 MIT Microsoft (markitdown contributors)
slides-nasa-clarreo slides pdf 15 public-domain-us-gov Yolanda L. Shea, Rajendra Bhatt / NASA Langley Research Center
slides-nist-pqc-update slides pdf 16 public-domain-us-gov Dustin Moody / NIST Computer Security Division
csv-census-popest-2023 spreadsheet csv – public-domain-us-gov U.S. Census Bureau, Population Division
csv-usgs-quakes-week spreadsheet csv – public-domain-us-gov U.S. Geological Survey, Earthquake Hazards Program
sheet-ers-crop-insurance spreadsheet xlsx – public-domain-us-gov USDA Economic Research Service
sheet-markitdown-xlsx spreadsheet xlsx – MIT Microsoft (markitdown contributors)
docx-hhs-market-research docx docx – public-domain-us-gov U.S. Department of Health and Human Services
docx-markitdown docx docx – MIT Microsoft (markitdown contributors)
docx-nsf-aaac-minutes docx docx – public-domain-us-gov U.S. National Science Foundation (AAAC)
epub-gutenberg-jekyll epub epub – public-domain Robert Louis Stevenson; Project Gutenberg
epub-gutenberg-relativity epub epub – public-domain Albert Einstein, translated by Robert W. Lawson; Project Gutenberg
epub-markitdown epub epub – MIT Microsoft (markitdown test suite)
html-bls-ooh-software-developers html html – public-domain-us-gov U.S. Bureau of Labor Statistics
html-markitdown-wikipedia html html – CC-BY-SA-4.0 (Wikipedia content in an MIT test file) Wikipedia contributors (article 'Microsoft'); Microsoft markitdown
html-wikipedia-roman-numerals html html – CC-BY-SA-4.0 Wikipedia contributors (article 'Roman numerals', oldid 1374259684)

Local doc-VLM resource measurements

Production worker run 36785961923, commit befd305, measured ten synthetic pages: English, Traditional Chinese and Simplified Chinese clean text, scans and photos, plus a mixed-language page. Each page starts a fresh worker, so timing includes model loading and process overhead. These are not OmniDocBench scores and are not directly comparable to the earlier 50-page spike.

Target / device Decoder weights Median seconds/page p90 seconds/page Peak process RSS MiB Mean CER
Linux x64 / CPU none 81.4 93.4 5320 1.90%
Linux x64 / CPU q8 229.7 280.1 4378 1.96%
Linux x64 / CPU q4 207.3 254.6 4186 2.41%
Linux arm64 / CPU none 108.2 129.5 5386 1.90%
Linux arm64 / CPU q8 59.8 73.4 4367 1.98%
Linux arm64 / CPU q4 62.4 77.3 4202 1.94%
macOS arm64 / Metal none 38.9 47.8 3763 1.88%
macOS arm64 / CPU q8 69.4 94.8 4477 1.94%
macOS arm64 / CPU q4 89.4 110.4 4458 2.41%
Windows x64 / CPU none 101.8 127.6 5358 1.90%
Windows x64 / CPU q8 254.2 338.0 4348 1.96%
Windows x64 / CPU q4 230.6 285.7 4175 2.47%

All twelve configurations converted all ten pages. Quantization applies only to the decoder's linear weights; vision, embeddings and layout stay unquantized. q8/q4 reduce peak RSS on Linux and Windows, but slow x64 CPUs substantially. They remain experimental, CPU-only options, not the default. macOS quantized rows use CPU rather than Metal. RSS was sampled every 100 ms (Windows uses the process peak working set), so it is not a whole-system or GPU-memory measurement.

The default remains unquantized. Automatic OCR never downloads models: without explicit setup it uses tesseract; anymd setup ocr opts into installed local models, including slower CPU inference. Native PDF text keeps its existing extraction path. The VLM binary adds 7.5 MiB on Linux x64, 6.1 MiB on Linux arm64, 6.9 MiB on macOS arm64 and 8.2 MiB on Windows x64 compared with the no-VLM build.

The Windows job passed tests, both binary builds and all measurements, then reached its two-hour limit while saving the Rust cache. Its measurements were uploaded before cancellation; the cancelled job itself is not a green release-target check.

OmniDocBench v1.6

OmniDocBench (CVPR 2025) is the public benchmark for turning page images into Markdown: 1,651 pages, scored on text, display formulas, tables, and reading order. We ran anymd on all 1,651 pages with the official end2end evaluator, pinned by commit. anymd read each page image with tesseract, the way it reads any scan or image: it returns the recognised text and nothing else. There is no layout model and no table, formula, or reading-order recognition, and tesseract runs with its default English model.

Overall Text (edit distance, lower is better) Formula (CDM) Table (TEDS) Reading order (edit distance, lower is better)
anymd 8.1.0 21.4 0.554 19.3 0.2 0.423

Overall is the leaderboard formula, ((1 - text edit distance) x 100 + table TEDS + formula CDM) / 3. 4 of the 1,651 pages hit anymd's 60 s OCR timeout, and 53 came back empty.

The scores published by the model authors, for context, are self-reported and come from vision-language models trained for this task:

Model Overall (self-reported)
TeleOCR (formerly NaviDC) 96.91
OvisOCR2 96.47
PaddleOCR-VL-1.6 96.34
GLM-OCR 95.22

This row measures anymd's scan and image path, which today is tesseract text recognition with no layout, table, or formula model. anymd's strength is documents that carry a text layer (PDF, Word, PowerPoint, Excel, EPUB, HTML), where it scores 96.4 on AgentDocBench above. The dataset ships page images only, so anymd's PDF engine is not measured here.

The dataset is licensed for research use only. We download it at run time, evaluate anymd, and publish our own score; nothing from the dataset is redistributed. The method, pins, and rerun command are in bench/omnidocbench. The scores above are the evaluator's own summary (run_summary.json).

Image and video extraction

Images and video require different extractors; there is no combined score. The image row below reuses the historical OmniDocBench v1.6 evaluation above, not a measurement of the current anymd OCR model. TeleOCR is the highest-scoring alternative in that v1.6 comparison (upstream leaderboard); its 96.91 is a published model-author result, not our same-run measurement.

For video embedded subtitles, the reference alternative is direct FFmpeg: it retains every authored cue in these fixtures and is the extractor anymd itself uses. The video rows are historical, same-run measurements of the published anymd 8.5.1 CLI and FFmpeg 6.1.1. They do not measure visual scene understanding or speech recognition. Speech-to-text below is a separate audio benchmark.

Dataset Samples Metric anymd Alternative Evidence
OmniDocBench v1.6 page images 1,651 pages Overall, higher is better 21.4 (8.1.0, English tesseract) TeleOCR 96.91 (author-reported) Image method and pins
MP4 mov_text subtitle fixtures 4 clips / 12 cues Exact text + start-second recall 91.7% (11/12) FFmpeg 100% (12/12) Raw video results
Matroska SubRip subtitle fixtures 4 clips / 12 cues Exact text + start-second recall 91.7% (11/12) FFmpeg 100% (12/12) Raw video results

The video datasets are small, MIT-licensed, authored regression fixtures: English, multiline text, CJK and repeated captions, muxed into three-second black videos. They are not representative film or lecture datasets. At 2026-10-08T02:07:30Z, both tools converted all eight clips successfully on the same Linux x86_64 Build lease. anymd retained 11 of 12 cues in each dataset, dropping the second identical caption at a different timestamp; FFmpeg retained all 12. Precision was 100% for both. That version's rolling-caption cleanup caused this loss; it is not a transcription error.

These dated results are not relabelled when subtitle rendering changes. The subtitle fidelity regression reruns the same authored corpus through the built CLI and MCP read, plus separate sidecar and edge-case fixtures. The native integration test writes a commit/hash receipt next to its tested binary; that is a correctness check, not a new timing benchmark. To keep a separate receipt, run:

python3 scripts/subtitle_fidelity.py --binary target/release/anymd --receipt target/subtitle-fidelity.json

Total extraction time was 2.108 s vs 1.128 s (anymd vs FFmpeg) for MP4 and 2.412 s vs 1.155 s for Matroska. Each tool ran once per clip, sequentially; times include process startup, exclude fixture generation, and are not throughput guarantees. anymd also probes metadata and renders Markdown; direct FFmpeg only extracts subtitles. The video benchmark method and rerun commands include exact versions, scoring rules, fixture text and all outputs. Cue end times, subsecond timestamps, frame OCR, metadata accuracy and audio transcription were not measured in these video rows.

Reproduce

python3 bench/fetch.py .cache/bench-corpus          # download and verify the corpus
cargo build --release -p anymd     # anymd
python3 -m venv .venv && .venv/bin/pip install -r bench/requirements/harness.txt

ANYMD_BIN=target/release/anymd .venv/bin/python bench/run.py --tool anymd \
  --corpus .cache/bench-corpus --out bench/results/anymd.json

To rerun the MarkItDown comparison, install its pinned dependencies in the same environment:

.venv/bin/pip install -r bench/requirements/markitdown.txt
.venv/bin/python bench/run.py --tool markitdown --corpus .cache/bench-corpus --out bench/results/markitdown.json

Print the leaderboard with .venv/bin/python bench/leaderboard.py --print.

The Benchmark workflow runs every tool on its own GitHub-hosted runner with the same system packages (poppler, tesseract with Chinese and Japanese, pandoc), and publishes the merged results and every Markdown output as an artifact.

Submit a tool

Add an adapter in bench/adapters/<tool>.py (a name, a version, and the command that prints Markdown), pin the tool in bench/requirements/<tool>.txt, add it to the workflow matrix, and open a pull request. The workflow runs the benchmark; commit the resulting bench/results/<tool>.json and the regenerated leaderboard. Details are in Submitting a tool.

The official @modelcontextprotocol/server-pdf is not included because it has no headless text path: it renders PDFs in an interactive viewer, and its read_pdf_bytes tool returns base64-encoded bytes.

Speech-to-text

We selected transcribe-cpp 0.2.4 / Qwen3-ASR-1.7B Q8_0 as the one local ASR runtime/model for every language. The crate built on all four benchmark targets: x86_64 Linux, aarch64 Linux, x86_64 macOS and x86_64 Windows. CrispASR failed on Linux arm64 and macOS x64; its optional standalone aligner is not required for ASR. Whisper appears below only as the historical benchmark control, not a shipping engine option.

Source: ASR run 36523821486, fixed-seed, revision-pinned samples; 200 utterances per Linux dataset. Qwen and whisper-turbo ran on CPU, four threads. Scores were recalculated from saved reference/hypothesis pairs, without rerunning inference.

Accuracy

Lower is better. Raw CER here means the existing baseline: NFKC, lowercase, punctuation/symbol/space/control stripping and OpenCC t2s on both sides. It is not punctuation-sensitive literal raw-text CER. English uses the pinned Open ASR Leaderboard EnglishTextNormalizer and WER.

Normalised (bracketed Latin glosses removed, numerals unified) applies the same transformations to references and hypotheses: remove Latin parenthetical annotations before stripping punctuation, then numeral ITN via cn2an 0.5.24 (zh/yue) or kanjize 1.6.1 (ja). Parentheses containing CJK text or just numbers remain. This is a scoring-sensitivity diagnostic, not a verified ground-truth correction: annotations may be spoken, and generic numeral conversion may alter lexical numerals. Kana/kanji readings are not collapsed. Normalisation can make scores worse as well as better.

Dataset Metric Qwen raw Qwen normalised (bracketed Latin glosses removed, numerals unified) whisper-turbo raw whisper-turbo normalised (same rules)
FLEURS en WER 4.54 4.54 6.33 6.33
FLEURS zh CER 7.73 2.53 9.21 5.03
FLEURS yue CER 5.41 3.89 12.10 12.29
FLEURS ja CER 5.93 5.85 4.80 4.65
LibriSpeech test-clean WER 2.00 2.00 2.23 2.23

Mandarin Qwen normalized CER has a 95% utterance-bootstrap CI of [1.92, 3.22]; whisper-turbo [4.30, 5.77]. Mandarin's original 602 errors include 372 deletions: removing Latin parenthetical annotations reduces deletions to 50 and total errors to 280; numeral ITN reduces total errors to 189. This explains most of the 7.73% raw score, rather than showing a model defect.

Accepted Japanese trade-off: on this 200-utterance FLEURS sample, Qwen trails whisper-turbo (5.93 vs 4.80 raw CER). Japanese numeral-only diagnostics give 5.79 vs 4.67; adding katakana-to-hiragana folding gives 5.56 vs 4.59 normalised CER. This separate diagnostic retains parenthetical annotations and the baseline OpenCC step; it is not the annotation-removal column above. Reproduce it with python3 bench-asr/report.py RESULTS --cjk-diagnostics --normalizer-dir NORMALIZERS using the saved run transcripts. Converting everything to kana readings is a different metric and still trails (4.50 vs 3.74); it cannot be presented as corrected CER. Real errors remain, including 地殻 → 近く and Taipei → 大阪. We accept this gap to use one model across languages.

No official-score reproduction claim. Qwen's published evaluation uses bfloat16, vLLM, greedy decoding, max_new_tokens=1024 and no language hint. The technical report uses CER for Mandarin and Cantonese but does not publish its complete text-normalisation protocol. Our CPU-quantized, language-hinted, sampled runs differ; the bootstrap intervals cover sample variation, not those protocol differences.

Runtime cost

On the full Linux samples transcribe-cpp's model-load-excluded RTF was 0.574 (en), 0.535 (zh), 0.559 (yue), 0.369 (ja), and 0.631 (LibriSpeech); RTF is compute time divided by audio time. See the original run's report for runner CPUs and peak memory; these are benchmark observations, not throughput guarantees.

The harness and all SHA-256/revision pins are in bench-asr/. report.py --cjk-diagnostics recalculates both scoring columns from saved summary.json / results.jsonl jobs. Its additional scoring dependencies are cn2an 0.5.24 and kanjize 1.6.1; the English normalizer is pinned in pins.json.