diff --git a/README.md b/README.md index b1ab46b..bc96110 100755 --- a/README.md +++ b/README.md @@ -5,11 +5,10 @@

-dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model +dots.ocr

-[![Blog](https://img.shields.io/badge/Blog-View_on_GitHub-333.svg?logo=github)](https://github.com/rednote-hilab/dots.ocr/blob/master/assets/blog.md) -[![HuggingFace](https://img.shields.io/badge/HuggingFace%20Weights-black.svg?logo=HuggingFace)](https://huggingface.co/rednote-hilab/dots.ocr) +[![HuggingFace](https://img.shields.io/badge/HuggingFace%20Weights-black.svg?logo=HuggingFace)](https://huggingface.co/rednote-hilab/dots.ocr-1.5) [![Arxiv](https://img.shields.io/badge/arXiv-Paper-B31B1B.svg?logo=arxiv)](https://arxiv.org/abs/2512.02498) @@ -26,974 +25,533 @@ dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model ## Introduction -**dots.ocr** is a powerful, multilingual document parser that unifies layout detection and content recognition within a single vision-language model while maintaining good reading order. Despite its compact 1.7B-parameter LLM foundation, it achieves state-of-the-art(SOTA) performance. - -1. **Powerful Performance:** **dots.ocr** achieves SOTA performance for text, tables, and reading order on [OmniDocBench](https://github.com/opendatalab/OmniDocBench), while delivering formula recognition results comparable to much larger models like Doubao-1.5 and gemini2.5-pro. -2. **Multilingual Support:** **dots.ocr** demonstrates robust parsing capabilities for low-resource languages, achieving decisive advantages across both layout detection and content recognition on our in-house multilingual documents benchmark. -3. **Unified and Simple Architecture:** By leveraging a single vision-language model, **dots.ocr** offers a significantly more streamlined architecture than conventional methods that rely on complex, multi-model pipelines. Switching between tasks is accomplished simply by altering the input prompt, proving that a VLM can achieve competitive detection results compared to traditional detection models like DocLayout-YOLO. -4. **Efficient and Fast Performance:** Built upon a compact 1.7B LLM, **dots.ocr** provides faster inference speeds than many other high-performing models based on larger foundations. - - -### Performance Comparison: dots.ocr vs. Competing Models - - -> **Notes:** -> - The EN, ZH metrics are the end2end evaluation results of [OmniDocBench](https://github.com/opendatalab/OmniDocBench), and Multilingual metric is the end2end evaluation results of dots.ocr-bench. - +**dots.ocr** Designed for universal accessibility, it possesses the capability to recognize virtually any human script. Beyond achieving state-of-the-art (SOTA) performance in standard multilingual document parsing among models of comparable size, dots.ocr-1.5 excels at converting structured graphics (e.g., charts and diagrams) directly into SVG code, parsing web screens and spotting scene text. ## News -* ```2025.10.31 ``` 🚀 We release [dots.ocr.base](https://huggingface.co/rednote-hilab/dots.ocr.base), foundation VLM focus on OCR tasks, also the base model of [dots.ocr](https://github.com/rednote-hilab/dots.ocr). Try it out! -* ```2025.07.30 ``` 🚀 We release [dots.ocr](https://github.com/rednote-hilab/dots.ocr), — a multilingual documents parsing model based on 1.7b llm, with SOTA performance. +* ```2026.2.16 ``` 🚀 We release [dots.ocr-1.5](https://huggingface.co/rednote-hilab/dots.ocr-1.5), trying to recognize any human scripts and symbols, not only the document parsing, but also the image parsing. We are simultaneously releasing [dots.ocr-1.5-svg](https://huggingface.co/rednote-hilab/dots.ocr-1.5-svg), which has more robust performance on image parsing +* ```2025.10.31 ``` 🚀 We release [dots.ocr.base](https://huggingface.co/rednote-hilab/dots.ocr.base), foundation VLM focus on OCR tasks, also the base model of [dots.ocr](https://huggingface.co/rednote-hilab/dots.ocr). Try it out! +* ```2025.07.30 ``` 🚀 We release [dots.ocr](https://huggingface.co/rednote-hilab/dots.ocr), — a multilingual documents parsing model based on 1.7b llm, with SOTA performance. -## Benchmark Results -### 1. OmniDocBench +## Evaluation -#### The end-to-end evaluation results of different tasks. +### 1. Document Parsing + +#### 1.1 Elo Score of different bench between latest models + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
modelsolmOCR-BenchOmniDocBench (v1.5)XDocParse
GLM-OCR859.9937.5742.1
PaddleOCR-VL-1.5873.6965.6797.6
HuanyuanOCR978.9974.4895.9
dots.ocr1027.4994.71133.4
dots.ocr-1.51089.01025.81157.1
Gemini 3 Pro1171.21102.11273.9
+ +> **Notes:** +> - Results for Gemini 3 Pro, PaddleOCR-VL-1.5, and GLM-OCR were obtained via APIs, while HuanyuanOCR results were generated using local inference. +> - The Elo score evaluation was conducted using Gemini 3 Flash. The prompt can be found at: [Elo Score Prompt](https://github.com/rednote-hilab/dots.ocr/blob/master/tools/elo_score_prompt.py). These results are consistent with the findings on [ocrarena](https://www.ocrarena.ai/battle). + + +#### 1.2 olmOCR-bench + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Model
Type		MethodsOverallEdit↓TextEdit↓FormulaEdit↓TableTEDS↑TableEdit↓Read OrderEdit↓
ENZHENZHENZHENZHENZHENZH
Pipeline
Tools		MinerU0.1500.3570.0610.2150.2780.57778.662.10.1800.3440.0790.292
Marker0.3360.5560.0800.3150.5300.88367.649.20.6190.6850.1140.340
Mathpix0.1910.3650.1050.3840.3060.45477.067.10.2430.3200.1080.304
Docling0.5890.9090.4160.9870.999161.325.00.6270.8100.3130.837
Pix2Text0.3200.5280.1380.3560.2760.61173.666.20.5840.6450.2810.499
Unstructured0.5860.7160.1980.4810.999100.0610.9980.1450.387
OpenParse0.6460.8140.6810.9740.996164.827.50.2840.6390.5950.641
PPStruct-V30.1450.2060.0580.0880.2950.535--0.1590.1090.0690.091
Expert
VLMs		GOT-OCR0.2870.4110.1890.3150.3600.52853.247.20.4590.5200.1410.280
Nougat0.4520.9730.3650.9980.4880.94139.900.5721.0000.3820.954
Mistral OCR0.2680.4390.0720.3250.3180.49575.863.60.6000.6500.0830.284
OLMOCR-sglang0.3260.4690.0970.2930.4550.65568.161.30.6080.6520.1450.277
SmolDocling-256M0.4930.8160.2620.8380.7530.99744.916.50.7290.9070.2270.522
Dolphin0.2060.3060.1070.1970.4470.58077.367.20.1800.2850.0910.162
MinerU 20.1390.2400.0470.1090.2970.53682.579.00.1410.1950.069<0.118
OCRFlux0.1950.2810.0640.1830.3790.61371.681.30.2530.1390.0860.187
MonkeyOCR-pro-3B0.1380.2060.0670.1070.2460.42181.587.50.1390.1110.1000.185
ArXivOld scans mathTablesOld scansHeaders & footersMulti columnLong tiny textBaseOverall
Mistral OCR API77.267.560.629.393.671.377.199.472.0±1.1
Marker 1.10.183.866.872.933.586.680.085.799.376.1±1.1
MinerU 2.5.4*76.654.684.933.796.678.283.593.775.2±1.1
DeepSeek-OCR77.273.680.233.396.166.479.499.875.7±1.0
Nanonets-OCR2-3B75.446.186.840.932.181.993.099.669.5±1.1
PaddleOCR-VL*85.771.084.137.897.079.985.798.580.0±1.0
Infinity-Parser 7B*84.483.885.047.988.784.286.499.882.5±?
olmOCR v0.4.083.082.384.947.796.183.781.999.782.4±1.1
Chandra OCR 0.1.0*82.280.388.050.490.881.292.399.983.1±0.9
dots.ocr82.164.288.340.994.182.481.299.579.1% ± 1.0%
dots.ocr-1.585.985.590.748.294.085.381.699.783.9% ± 0.9
-General
VLMs -GPT4o -0.233 -0.399 -0.144 -0.409 -0.425 -0.606 -72.0 -62.9 -0.234 -0.329 -0.128 -0.251 - + + + +> **Note:** +> - The metrics are from [olmocr](https://github.com/allenai/olmocr), and our own internal evaluations. +> - We delete the Page-header and Page-footer cells in the result markdown. + + +#### 1.3 Other Benchmarks + + + - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - + + + + + - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + + + + + + - + + - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Qwen2-VL-72B0.2520.3270.0960.2180.4040.48776.876.40.3870.4080.1190.193Model TypeMethodsSizeOmniDocBench(v1.5)
TextEdit↓		OmniDocBench(v1.5)
Read OrderEdit↓		pdf-parse-bench
GeneralVLMsGemini-2.5 Pro-0.0750.0979.06
Qwen2.5-VL-72B0.2140.2610.0920.180.3150.43482.983.90.3410.2620.1060.168Qwen3-VL-235B-A22B-Instruct235B0.0690.0689.71
Gemini2.5-Pro0.1480.2120.0550.1680.3560.43985.886.40.130.1190.0490.121gemini3pro-0.0660.0799.68
SpecializedVLMsMistral OCR-0.1640.1448.84
doubao-1-5-thinking-vision-pro-2504280.1400.1620.0430.0850.2950.38483.389.30.1650.085Deepseek-OCR3B0.0730.0868.26
MonkeyOCR-3B3B0.0750.1299.27
OCRVerse4B 0.0580.0940.071--
Expert VLMsdots.ocr0.1250.1600.0320.0660.3290.41688.689.00.0990.0920.0400.067
MonkeyOCR-pro-3B3B0.0750.128-
MinerU2.51.2B0.0470.044-
PaddleOCR-VL0.9B0.0350.0439.51
HunyuanOCR0.9B0.042--
PaddleOCR-VL1.50.9B0.0350.042-
GLMOCR0.9B0.040.043-
dots.ocr3B0.0480.0539.29
dots.ocr-1.53B0.0310.0299.54
+> **Note:** +> - Metrics are sourced from [OmniDocBench](https://github.com/opendatalab/OmniDocBench) and other model publications. [pdf-parse-bench](https://github.com/phorn1/pdf-parse-bench) results are reproduced by Qwen3-VL-235B-A22B-Instruct. +> - Formula and Table metrics for OmniDocBench1.5 are omitted due to their high sensitivity to detection and matching protocols. -#### The end-to-end text recognition performance across 9 PDF page types. + +### 2. Vision-Language Parsing +Visual languages (e.g., charts, graphics, chemical formulas, logos) encapsulate dense human knowledge. **dots.ocr-1.5** unifies the interpretation of these elements by parsing them directly into **SVG code**. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Model
Type		ModelsBookSlidesFinancial
Report		TextbookExam
Paper		MagazineAcademic
Papers		NotesNewspaperOverall
Pipeline
Tools		MinerU0.0550.1240.0330.1020.1590.0720.0250.9840.1710.206
Marker0.0740.3400.0890.3190.4520.1530.0590.6510.1920.274
Mathpix0.1310.2200.2020.2160.2780.1470.0910.6340.6900.300
Expert
VLMs		GOT-OCR0.1110.2220.0670.1320.2040.1980.1790.3880.7710.267
Nougat0.7340.9581.0000.8200.9300.8300.2140.9910.8710.806
Dolphin0.0910.1310.0570.1460.2310.1210.0740.3630.3070.177
OCRFlux0.0680.1250.0920.1020.1190.0830.0470.2230.5360.149
MonkeyOCR-pro-3B0.0840.1290.0600.0900.1070.0730.0500.1710.1070.100
General
VLMs		GPT4o0.1570.1630.3480.1870.2810.1730.1460.6070.7510.316
Qwen2.5-VL-7B0.1480.0530.1110.1370.1890.1170.1340.2040.7060.205
InternVL3-8B0.1630.0560.1070.1090.1290.1000.1590.1500.6810.188
doubao-1-5-thinking-vision-pro-2504280.0480.0480.0240.0620.0850.0510.0390.0960.1810.073
Expert VLMsdots.ocr0.0310.0470.0110.0820.0790.0280.0290.1090.0560.055
- -> **Notes:** -> - The metrics are from [MonkeyOCR](https://github.com/Yuliang-Liu/MonkeyOCR), [OmniDocBench](https://github.com/opendatalab/OmniDocBench), and our own internal evaluations. -> - We delete the Page-header and Page-footer cells in the result markdown. -> - We use tikz_preprocess pipeline to upsample the images to dpi 200. - - -### 2. **dots.ocr-bench** - -This is an inhouse benchmark which contain 1493 pdf images with 100 languages. - -#### The end-to-end evaluation results of different tasks. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MethodsOverallEdit↓TextEdit↓FormulaEdit↓TableTEDS↑TableEdit↓Read OrderEdit↓
MonkeyOCR-3B0.4830.4450.62750.930.4520.409
doubao-1-5-thinking-vision-pro-2504280.2910.2260.44071.20.2600.238
doubao-1-60.2990.2700.41771.00.2580.253
Gemini2.5-Pro0.2510.1630.40277.10.2360.202
dots.ocr 0.1770.0750.29779.20.1860.152
- -> **Notes:** -> - We use the same metric calculation pipeline of [OmniDocBench](https://github.com/opendatalab/OmniDocBench). -> - We delete the Page-header and Page-footer cells in the result markdown. - -#### Layout Detection - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MethodF1@IoU=.50:.05:.95↑F1@IoU=.50↑
OverallTextFormulaTablePictureOverallTextFormulaTablePicture
DocLayout-YOLO-DocStructBench0.7330.6940.4800.8030.6190.8060.7790.6200.8580.678
dots.ocr-parse all0.8310.8010.6540.8380.7480.9220.9090.7700.8880.831
dots.ocr-detection only 0.8450.8160.7160.8750.7650.9300.9170.8320.9180.843
- -> **Notes:** -> - prompt_layout_all_en for **parse all**, prompt_layout_only_en for **detection only**, please refer to [prompts](https://github.com/rednote-hilab/dots.ocr/blob/master/dots_ocr/utils/prompts.py) - - -### 3. olmOCR-bench. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelArXivOld Scans
Math		TablesOld ScansHeaders and
Footers		Multi
column		Long Tiny
Text		BaseOverall
GOT OCR52.752.00.222.193.642.029.994.048.3 ± 1.1
Marker76.057.957.627.884.972.984.699.170.1 ± 1.1
MinerU75.447.460.917.396.659.039.196.661.5 ± 1.1
Mistral OCR77.267.560.629.393.671.377.199.472.0 ± 1.1
Nanonets OCR67.068.677.739.540.769.953.499.364.5 ± 1.1
GPT-4o
(No Anchor)		51.575.569.140.994.268.954.196.768.9 ± 1.1
GPT-4o
(Anchored)		53.574.570.040.793.869.360.696.869.9 ± 1.1
Gemini Flash 2
(No Anchor)		32.156.361.427.848.058.784.494.057.8 ± 1.1
Gemini Flash 2
(Anchored)		54.556.172.134.264.761.571.595.663.8 ± 1.2
Qwen 2 VL
(No Anchor)		19.731.724.217.188.98.36.855.531.5 ± 0.9
Qwen 2.5 VL
(No Anchor)		63.165.767.338.673.668.349.198.365.5 ± 1.2
olmOCR v0.1.75
(No Anchor)		71.571.471.442.894.177.771.097.874.7 ± 1.1
olmOCR v0.1.75
(Anchored)		74.971.271.042.294.578.373.398.375.5 ± 1.0
MonkeyOCR-pro-3B83.868.874.636.191.276.680.195.375.8 ± 1.0
dots.ocr82.164.288.340.994.182.481.299.579.1 ± 1.0
MethodsUnisvgChartmimicDesign2CodeGenexamSciGenChemDraw
Low-LevelHigh-LevelScore
OCRVerse0.6320.8520.7630.799---0.881
Gemini 3 Pro0.5630.8500.7350.7880.7600.7560.7830.839
dots.ocr-1.50.8500.9230.8940.7720.8010.6640.6600.790
dots.ocr-1.5-svg0.8600.9310.9020.9050.8340.80.7970.901
> **Note:** -> - The metrics are from [MonkeyOCR](https://github.com/Yuliang-Liu/MonkeyOCR), -[olmocr](https://github.com/allenai/olmocr), and our own internal evaluations. -> - We delete the Page-header and Page-footer cells in the result markdown. +> - We use the ISVGEN metric from [UniSVG](https://ryanlijinke.github.io/) to evaluate the parsing result. For benchmarks that do not natively support image parsing, we use the original images as input, and calculate the ISVGEN score between the rendered output and the original image. +> - [OCRVerse](https://github.com/DocTron-hub/OCRVerse) results are derived from various code formats (e.g., SVG, Python), whereas results for Gemini 3 Pro and dots.ocr-1.5 are based specifically on SVG code. +> - Due to the capacity constraints of a 3B-parameter VLM, dots.ocr-1.5 may not excel in all tasks yet like svg. To complement this, we are simultaneously releasing dots.ocr-1.5-svg. We plan to further address these limitations in future updates. + + +### 3. General Vision Tasks + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelCharXiv_descriptiveCharXiv_reasoningOCR_ReasoninginfovqadocvqaChartQAOCRBenchAI2DCountBenchQArefcoco
Qwen3vl-2b-instruct62.326.8-72.493.3-85.876.988.4-
dots.ocr-1.577.455.322.8573.7691.8583.286.082.1694.4680.03
# Quick Start ## 1. Installation -### Install dots.ocr +### Install dots.ocr-1.5 ```shell conda create -n dots_ocr python=3.12 conda activate dots_ocr @@ -1015,7 +573,7 @@ pip install -e . ### Download Model Weights -> 💡**Note:** Please use a directory name without periods (e.g., `DotsOCR` instead of `dots.ocr`) for the model save path. This is a temporary workaround pending our integration with Transformers. +> 💡**Note:** Please use a directory name without periods (e.g., `DotsOCR_1_5` instead of `dots.ocr-1.5`) for the model save path. This is a temporary workaround pending our integration with Transformers. ```shell python3 tools/download_model.py @@ -1028,14 +586,30 @@ python3 tools/download_model.py --type modelscope ### vLLM inference We highly recommend using vLLM for deployment and inference. All of our evaluations results are based on vLLM 0.9.1 via out-of-tree model registration. **Since vLLM version 0.11.0, Dots OCR has been officially integrated into vLLM with verified performance** and you can use vLLM docker image directly (e.g, `vllm/vllm-openai:v0.11.0`) to deploy the model server. +> **Note:** +> - We found a little bit performance drop when using vLLM 0.11.0. We are working on a fix. + ```shell # Launch vLLM model server -vllm serve rednote-hilab/dots.ocr --trust-remote-code --async-scheduling --gpu-memory-utilization 0.95 +## dots.ocr-1.5 +CUDA_VISIBLE_DEVICES=0 vllm serve rednote-hilab/dots.ocr-1.5 --tensor-parallel-size 1 --gpu-memory-utilization 0.9 --chat-template-content-format string --served-model-name model --trust-remote-code + +## dots.ocr-1.5-svg +CUDA_VISIBLE_DEVICES=0 vllm serve rednote-hilab/dots.ocr-1.5-svg --tensor-parallel-size 1 --gpu-memory-utilization 0.9 --chat-template-content-format string --served-model-name model --trust-remote-code # vLLM API Demo # See dots_ocr/model/inference.py for details on parameter and prompt settings # that help achieve the best output quality. +## document parsing python3 ./demo/demo_vllm.py --prompt_mode prompt_layout_all_en +## web parsing + +## scene spoting + +## image parsing with svg code + +## general qa + ``` ### Hugginface inference @@ -1052,7 +626,7 @@ from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer from qwen_vl_utils import process_vision_info from dots_ocr.utils import dict_promptmode_to_prompt -model_path = "./weights/DotsOCR" +model_path = "./weights/DotsOCR_1_5" model = AutoModelForCausalLM.from_pretrained( model_path, attn_implementation="flash_attention_2", @@ -1146,8 +720,6 @@ python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_layout_only_en # Parse text only, except Page-header and Page-footer python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_ocr -# Parse layout info by bbox -python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_grounding_ocr --bbox 163 241 1536 705 ``` **Based on Transformers**, you can parse an image or a pdf file using the same commands above, just add `--use_hf true`. @@ -1164,62 +736,48 @@ python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_grounding_ocr -- + ## 4. Demo -You can run the demo with the following command, or try directly at [live demo](https://dotsocr.xiaohongshu.com/) -```bash -python demo/demo_gradio.py -``` - -We also provide a demo for grounding ocr: -```bash -python demo/demo_gradio_annotion.py -``` +Have fun with the [live demo](https://dotsocr.xiaohongshu.com/). -### Example for formula document +### Examples for document parsing formula1.png -formula2.png -formula3.png - -### Example for table document -table1.png -table2.png table3.png - -### Example for multilingual document Tibetan.png tradition_zh.png nl.png kannada.png russian.png -### Example for reading order -reading_order.png -### Example for grounding ocr -grounding.png +### Examples for image parsing +svg_1.png +svg_2.png +svg_4.png +svg_5.png +svg_6.png +> **Note:** +> - Inferenced by dots.ocr-1.5-svg -# Acknowledgments -We would like to thank [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL), [aimv2](https://github.com/apple/ml-aim), [MonkeyOCR](https://github.com/Yuliang-Liu/MonkeyOCR), -[OmniDocBench](https://github.com/opendatalab/OmniDocBench), [PyMuPDF](https://github.com/pymupdf/PyMuPDF), for providing code and models. +### Example for web parsing +webpage_1.png +webpage_2.png + +### Examples for scene spotting +scene_1.png +scene_2.png -We also thank [DocLayNet](https://github.com/DS4SD/DocLayNet), [M6Doc](https://github.com/HCIILAB/M6Doc), [CDLA](https://github.com/buptlihang/CDLA), [D4LA](https://github.com/AlibabaResearch/AdvancedLiterateMachinery) for providing valuable datasets. # Limitation & Future Work - **Complex Document Elements:** - - **Table&Formula**: dots.ocr is not yet perfect for high-complexity tables and formula extraction. - - **Picture**: Pictures in documents are currently not parsed. + - **Table&Formula**: The extraction of complex tables and mathematical formulas persists as a difficult task given the model's compact architecture. + - **Picture**: We have adopted an SVG code representation for parsing structured graphics; however, the performance has yet to achieve the desired level of robustness. -- **Parsing Failures:** The model may fail to parse under certain conditions: - - When the character-to-pixel ratio is excessively high. Try enlarging the image or increasing the PDF parsing DPI (a setting of 200 is recommended). However, please note that the model performs optimally on images with a resolution under 11289600 pixels. - - Continuous special characters, such as ellipses (`...`) and underscores (`_`), may cause the prediction output to repeat endlessly. In such scenarios, consider using alternative prompts like `prompt_layout_only_en`, `prompt_ocr`, or `prompt_grounding_ocr` ([details here](https://github.com/rednote-hilab/dots.ocr/blob/master/dots_ocr/utils/prompts.py)). - -- **Performance Bottleneck:** Despite its 1.7B parameter LLM foundation, **dots.ocr** is not yet optimized for high-throughput processing of large PDF volumes. +- **Parsing Failures:** While we have reduced the rate of parsing failures compared to the previous version, these issues may still occur occasionally. We remain committed to further resolving these edge cases in future updates. -We are committed to achieving more accurate table and formula parsing, as well as enhancing the model's OCR capabilities for broader generalization, all while aiming for **a more powerful, more efficient model**. Furthermore, we are actively considering the development of **a more general-purpose perception model** based on Vision-Language Models (VLMs), which would integrate general detection, image captioning, and OCR tasks into a unified framework. **Parsing the content of the pictures in the documents** is also a key priority for our future work. -We believe that collaboration is the key to tackling these exciting challenges. If you are passionate about advancing the frontiers of document intelligence and are interested in contributing to these future endeavors, we would love to hear from you. Please reach out to us via email at: [yanqing4@xiaohongshu.com]. # Citation diff --git a/README_hf.md b/README_hf.md new file mode 100755 index 0000000..a43b508 --- /dev/null +++ b/README_hf.md @@ -0,0 +1,778 @@ +--- +license: mit +library_name: dots_ocr_1_5 +pipeline_tag: image-text-to-text +tags: +- image-to-text +- ocr +- document-parse +- layout +- table +- formula +- transformers +- custom_code +language: +- en +- zh +- multilingual +--- + +
+ +

+ +

+ +

+dots.ocr-1.5: Recognize Any Human Scripts and Symbols +

+ +[![HuggingFace](https://img.shields.io/badge/HuggingFace%20Weights-black.svg?logo=HuggingFace)](https://huggingface.co/rednote-hilab/dots.ocr-1.5) + + +
+ đŸ–„ïž Live Demo | + 💬 WeChat | + 📕 rednote +
+ +
+ + + +## Introduction + +We present **dots.ocr-1.5**, a 3B-parameter multimodal model composed of a 1.2B vision encoder and a 1.7B language model. Designed for universal accessibility, it possesses the capability to recognize virtually any human script. Beyond achieving state-of-the-art (SOTA) performance in standard multilingual document parsing among models of comparable size, dots.ocr-1.5 excels at converting structured graphics (e.g., charts and diagrams) directly into SVG code, parsing web screens and spotting scene text. Furthermore, the model demonstrates competitive performance in general OCR, object grounding & counting tasks. + +1. **Stronger Document Parsing Performance:** dots.ocr-1.5 maintains SOTA performance among latest OCR models, particularly on **multilingual documents**. Addressing the significant bias inherent in the detection & matching rules of certain benchmarks —which often fail to accurately reflect a model's true capabilities—we adopted an **Elo score** evaluation system. Under this metric, the performance landscape shifts significantly, highlighting the superior robustness of our model compared to conventional rankings. +2. **Unified Vision-Language Parsing**: Visual languages (e.g., charts, graphics, chemical formulas, logos) encapsulate dense human knowledge, akin to natural language. dots.ocr-1.5 unifies the interpretation of these elements by parsing them directly into SVG code. We have validated the effectiveness of this approach, demonstrating impressive results in structural and semantic recognition. +3. **Broader and More General Capabilities**: Compared to dots.ocr, dots.ocr-1.5 supports a significantly wider array of tasks. It extends beyond standard OCR to handle web screen parsing, scene text spotting, object grounding & counting, and other general OCR QA tasks. + + +## Evaluation + +### 1. Document Parsing + +#### 1.1 Elo Score of different bench between latest models + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
modelsolmOCR-BenchOmniDocBench (v1.5)XDocParse
GLM-OCR859.9937.5742.1
PaddleOCR-VL-1.5873.6965.6797.6
HuanyuanOCR978.9974.4895.9
dots.ocr1027.4994.71133.4
dots.ocr-1.51089.01025.81157.1
Gemini 3 Pro1171.21102.11273.9
+ +> **Notes:** +> - Results for Gemini 3 Pro, PaddleOCR-VL-1.5, and GLM-OCR were obtained via APIs, while HuanyuanOCR results were generated using local inference. +> - The Elo score evaluation was conducted using Gemini 3 Flash. The prompt can be found at: [Elo Score Prompt](https://github.com/rednote-hilab/dots.ocr/blob/master/tools/elo_score_prompt.py). These results are consistent with the findings on [ocrarena](https://www.ocrarena.ai/battle). + + +#### 1.2 olmOCR-bench + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ArXivOld scans mathTablesOld scansHeaders & footersMulti columnLong tiny textBaseOverall
Mistral OCR API77.267.560.629.393.671.377.199.472.0±1.1
Marker 1.10.183.866.872.933.586.680.085.799.376.1±1.1
MinerU 2.5.4*76.654.684.933.796.678.283.593.775.2±1.1
DeepSeek-OCR77.273.680.233.396.166.479.499.875.7±1.0
Nanonets-OCR2-3B75.446.186.840.932.181.993.099.669.5±1.1
PaddleOCR-VL*85.771.084.137.897.079.985.798.580.0±1.0
Infinity-Parser 7B*84.483.885.047.988.784.286.499.882.5±?
olmOCR v0.4.083.082.384.947.796.183.781.999.782.4±1.1
Chandra OCR 0.1.0*82.280.388.050.490.881.292.399.983.1±0.9
dots.ocr82.164.288.340.994.182.481.299.579.1% ± 1.0%
dots.ocr-1.585.985.590.748.294.085.381.699.783.9% ± 0.9
+ + + + +> **Note:** +> - The metrics are from [olmocr](https://github.com/allenai/olmocr), and our own internal evaluations. +> - We delete the Page-header and Page-footer cells in the result markdown. + + +#### 1.3 Other Benchmarks + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Model TypeMethodsSizeOmniDocBench(v1.5)
TextEdit↓		OmniDocBench(v1.5)
Read OrderEdit↓		pdf-parse-bench
GeneralVLMsGemini-2.5 Pro-0.0750.0979.06
Qwen3-VL-235B-A22B-Instruct235B0.0690.0689.71
gemini3pro-0.0660.0799.68
SpecializedVLMsMistral OCR-0.1640.1448.84
Deepseek-OCR3B0.0730.0868.26
MonkeyOCR-3B3B0.0750.1299.27
OCRVerse4B0.0580.071--
MonkeyOCR-pro-3B3B0.0750.128-
MinerU2.51.2B0.0470.044-
PaddleOCR-VL0.9B0.0350.0439.51
HunyuanOCR0.9B0.042--
PaddleOCR-VL1.50.9B0.0350.042-
GLMOCR0.9B0.040.043-
dots.ocr3B0.0480.0539.29
dots.ocr-1.53B0.0310.0299.54
+ +> **Note:** +> - Metrics are sourced from [OmniDocBench](https://github.com/opendatalab/OmniDocBench) and other model publications. [pdf-parse-bench](https://github.com/phorn1/pdf-parse-bench) results are reproduced by Qwen3-VL-235B-A22B-Instruct. +> - Formula and Table metrics for OmniDocBench1.5 are omitted due to their high sensitivity to detection and matching protocols. + + +### 2. Vision-Language Parsing +Visual languages (e.g., charts, graphics, chemical formulas, logos) encapsulate dense human knowledge. **dots.ocr-1.5** unifies the interpretation of these elements by parsing them directly into **SVG code**. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MethodsUnisvgChartmimicDesign2CodeGenexamSciGenChemDraw
Low-LevelHigh-LevelScore
OCRVerse0.6320.8520.7630.799---0.881
Gemini 3 Pro0.5630.8500.7350.7880.7600.7560.7830.839
dots.ocr-1.50.8500.9230.8940.7720.8010.6640.6600.790
dots.ocr-1.5-svg0.8600.9310.9020.9050.8340.80.7970.901
+ + +> **Note:** +> - We use the ISVGEN metric from [UniSVG](https://ryanlijinke.github.io/) to evaluate the parsing result. For benchmarks that do not natively support image parsing, we use the original images as input, and calculate the ISVGEN score between the rendered output and the original image. +> - [OCRVerse](https://github.com/DocTron-hub/OCRVerse) results are derived from various code formats (e.g., SVG, Python), whereas results for Gemini 3 Pro and dots.ocr-1.5 are based specifically on SVG code. +> - Due to the capacity constraints of a 3B-parameter VLM, dots.ocr-1.5 may not excel in all tasks yet like svg. To complement this, we are simultaneously releasing dots.ocr-1.5-svg. We plan to further address these limitations in future updates. + + +### 3. General Vision Tasks + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelCharXiv_descriptiveCharXiv_reasoningOCR_ReasoninginfovqadocvqaChartQAOCRBenchAI2DCountBenchQArefcoco
Qwen3vl-2b-instruct62.326.8-72.493.3-85.876.988.4-
dots.ocr-1.577.455.322.8573.7691.8583.286.082.1694.4680.03
+ + + +# Quick Start +## 1. Installation +### Install dots.ocr-1.5 +```shell +conda create -n dots_ocr python=3.12 +conda activate dots_ocr + +git clone https://github.com/rednote-hilab/dots.ocr.git +cd dots.ocr + +# Install pytorch, see https://pytorch.org/get-started/previous-versions/ for your cuda version +pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 --index-url https://download.pytorch.org/whl/cu128 +pip install -e . +``` + +If you have trouble with the installation, try our [Docker Image](https://hub.docker.com/r/rednotehilab/dots.ocr) for an easier setup, and follow these steps: +```shell +git clone https://github.com/rednote-hilab/dots.ocr.git +cd dots.ocr +pip install -e . +``` + + +### Download Model Weights +> 💡**Note:** Please use a directory name without periods (e.g., `DotsOCR_1_5` instead of `dots.ocr-1.5`) for the model save path. This is a temporary workaround pending our integration with Transformers. +```shell +python3 tools/download_model.py +``` + + +## 2. Deployment +### vLLM inference +We highly recommend using vllm for deployment and inference. + +```shell +# launch vllm server +## dots.ocr-1.5 +CUDA_VISIBLE_DEVICES=0 vllm serve rednote-hilab/dots.ocr-1.5 --tensor-parallel-size 1 --gpu-memory-utilization 0.9 --chat-template-content-format string --served-model-name model --trust-remote-code + +## dots.ocr-1.5-svg +CUDA_VISIBLE_DEVICES=0 vllm serve rednote-hilab/dots.ocr-1.5-svg --tensor-parallel-size 1 --gpu-memory-utilization 0.9 --chat-template-content-format string --served-model-name model --trust-remote-code + +# vllm api demo +## document parsing +python3 ./demo/demo_vllm.py --prompt_mode prompt_layout_all_en +## web parsing + +## scene spoting + +## image parsing with svg code + +## general qa + +``` + +### Hugginface inference +```shell +python3 demo/demo_hf.py +``` + +
+Hugginface inference details + +```python +import torch +from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer +from qwen_vl_utils import process_vision_info +from dots_ocr.utils import dict_promptmode_to_prompt + +model_path = "./weights/DotsOCR_1_5" +model = AutoModelForCausalLM.from_pretrained( + model_path, + attn_implementation="flash_attention_2", + torch_dtype=torch.bfloat16, + device_map="auto", + trust_remote_code=True +) +processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True) + +image_path = "demo/demo_image1.jpg" +prompt = """Please output the layout information from the PDF image, including each layout element's bbox, its category, and the corresponding text content within the bbox. + +1. Bbox format: [x1, y1, x2, y2] + +2. Layout Categories: The possible categories are ['Caption', 'Footnote', 'Formula', 'List-item', 'Page-footer', 'Page-header', 'Picture', 'Section-header', 'Table', 'Text', 'Title']. + +3. Text Extraction & Formatting Rules: + - Picture: For the 'Picture' category, the text field should be omitted. + - Formula: Format its text as LaTeX. + - Table: Format its text as HTML. + - All Others (Text, Title, etc.): Format their text as Markdown. + +4. Constraints: + - The output text must be the original text from the image, with no translation. + - All layout elements must be sorted according to human reading order. + +5. Final Output: The entire output must be a single JSON object. +""" + +messages = [ + { + "role": "user", + "content": [ + { + "type": "image", + "image": image_path + }, + {"type": "text", "text": prompt} + ] + } + ] + +# Preparation for inference +text = processor.apply_chat_template( + messages, + tokenize=False, + add_generation_prompt=True +) +image_inputs, video_inputs = process_vision_info(messages) +inputs = processor( + text=[text], + images=image_inputs, + videos=video_inputs, + padding=True, + return_tensors="pt", +) + +inputs = inputs.to("cuda") + +# Inference: Generation of the output +generated_ids = model.generate(**inputs, max_new_tokens=24000) +generated_ids_trimmed = [ + out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) +] +output_text = processor.batch_decode( + generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False +) +print(output_text) + +``` + +
+ +## 3. Document Parse +**Based on vLLM server**, you can parse an image or a pdf file using the following commands: +```bash + +# Parse all layout info, both detection and recognition +# Parse a single image +python3 dots_ocr/parser.py demo/demo_image1.jpg +# Parse a single PDF +python3 dots_ocr/parser.py demo/demo_pdf1.pdf --num_thread 64 # try bigger num_threads for pdf with a large number of pages + +# Layout detection only +python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_layout_only_en + +# Parse text only, except Page-header and Page-footer +python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_ocr + + +``` + +
+Output Results + +1. **Structured Layout Data** (`demo_image1.json`): A JSON file containing the detected layout elements, including their bounding boxes, categories, and extracted text. +2. **Processed Markdown File** (`demo_image1.md`): A Markdown file generated from the concatenated text of all detected cells. + * An additional version, `demo_image1_nohf.md`, is also provided, which excludes page headers and footers for compatibility with benchmarks like Omnidocbench and olmOCR-bench. +3. **Layout Visualization** (`demo_image1.jpg`): The original image with the detected layout bounding boxes drawn on it. + +
+ +## 4. Demo +Have fun with the [live demo](https://dotsocr.xiaohongshu.com/). + + +### Examples for document parsing +formula1.png +table3.png +Tibetan.png +tradition_zh.png +nl.png +kannada.png +russian.png + + +### Examples for image parsing +svg_1.png +svg_2.png +svg_4.png +svg_5.png +svg_6.png + +> **Note:** +> - Inferenced by dots.ocr-1.5-svg + +### Example for web parsing +webpage_1.png +webpage_2.png + +### Examples for scene spotting +scene_1.png +scene_2.png + + + +## Limitation & Future Work + +- **Complex Document Elements:** + - **Table&Formula**: The extraction of complex tables and mathematical formulas persists as a difficult task given the model's compact architecture. + - **Picture**: We have adopted an SVG code representation for parsing structured graphics; however, the performance has yet to achieve the desired level of robustness. + +- **Parsing Failures:** While we have reduced the rate of parsing failures compared to the previous version, these issues may still occur occasionally. We remain committed to further resolving these edge cases in future updates. \ No newline at end of file diff --git a/assets/showcase_dots_ocr_1_5/result/scene_1.png b/assets/showcase_dots_ocr_1_5/result/scene_1.png new file mode 100644 index 0000000..f5f3a57 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/scene_1.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/scene_2.png b/assets/showcase_dots_ocr_1_5/result/scene_2.png new file mode 100644 index 0000000..a8031c6 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/scene_2.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/svg_1.png b/assets/showcase_dots_ocr_1_5/result/svg_1.png new file mode 100644 index 0000000..5b8c1f3 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/svg_1.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/svg_2.png b/assets/showcase_dots_ocr_1_5/result/svg_2.png new file mode 100644 index 0000000..65b64a7 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/svg_2.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/svg_4.png b/assets/showcase_dots_ocr_1_5/result/svg_4.png new file mode 100644 index 0000000..c458456 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/svg_4.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/svg_5.png b/assets/showcase_dots_ocr_1_5/result/svg_5.png new file mode 100644 index 0000000..fe8d103 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/svg_5.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/svg_6.png b/assets/showcase_dots_ocr_1_5/result/svg_6.png new file mode 100644 index 0000000..803fcd4 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/svg_6.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/webpage_1.png b/assets/showcase_dots_ocr_1_5/result/webpage_1.png new file mode 100644 index 0000000..d2c9042 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/webpage_1.png differ diff --git a/assets/showcase_dots_ocr_1_5/result/webpage_2.png b/assets/showcase_dots_ocr_1_5/result/webpage_2.png new file mode 100644 index 0000000..a7da568 Binary files /dev/null and b/assets/showcase_dots_ocr_1_5/result/webpage_2.png differ diff --git a/requirements.txt b/requirements.txt index 7eed6f1..15852ca 100755 --- a/requirements.txt +++ b/requirements.txt @@ -7,5 +7,6 @@ qwen_vl_utils transformers==4.51.3 huggingface_hub modelscope -flash-attn==2.8.0.post2 +# flash-attn==2.8.0.post2 # to speed up inference need flash-attn accelerate +cairosvg \ No newline at end of file diff --git a/tools/download_model.py b/tools/download_model.py index 32d7087..d5db841 100755 --- a/tools/download_model.py +++ b/tools/download_model.py @@ -5,11 +5,11 @@ import os if __name__ == '__main__': parser = ArgumentParser() parser.add_argument('--type', '-t', type=str, default="huggingface") - parser.add_argument('--name', '-n', type=str, default="rednote-hilab/dots.ocr") + parser.add_argument('--name', '-n', type=str, default="rednote-hilab/dots.ocr-1.5") args = parser.parse_args() script_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) print(f"Attention: The model save dir dots.ocr should be replace by a name without `.` like DotsOCR, util we merge our code to transformers.") - model_dir = os.path.join(script_dir, "weights/DotsOCR") + model_dir = os.path.join(script_dir, "weights/DotsOCR_1_5") if not os.path.exists(model_dir): os.makedirs(model_dir) if args.type == "huggingface": diff --git a/tools/elo_score_prompt.py b/tools/elo_score_prompt.py new file mode 100644 index 0000000..c8b5e06 --- /dev/null +++ b/tools/elo_score_prompt.py @@ -0,0 +1,89 @@ +def construct_prompt(c1_text, c2_text): + """ + Constructs the complete Prompt sent to Gemini (English Version). + c1_text: Markdown text from Model 1 + c2_text: Markdown text from Model 2 + """ + + prompt = f"""You are an expert in evaluating OCR content accuracy. Please compare the model outputs with the original image, focusing heavily on **content accuracy** while ignoring formatting and layout differences. + +【Evaluation Focus - Focus ONLY on Content Accuracy】 +1. **Text Accuracy**: + - Typos: Character recognition errors (e.g., "test" recognized as "tost"). + - Omissions: Missing characters or words present in the original text. + - Hallucinations: Adding characters that do not exist in the original text. + +2. **Table Accuracy**: + - Correctness of data and text within the table. + - Completeness of cell content. + - Correct row/column alignment. + +3. **Formula Accuracy** (Evaluate based on): + - **Correctness**: Are mathematical symbols, variables, and operators preserved accurately? + - **Completeness**: Are all parts of the formula present without omission? + - **Semantic Equivalence**: Does the extracted formula convey the exact same mathematical meaning? + +【Tie Judgment Criteria - Important】 +You must judge as a **tie** in the following cases: +- Text content is identical, differing only in Markdown formatting. +- Table data is identical, differing only in Markdown table syntax. +- Formula content is semantically equivalent, differing only in LaTeX representation. +- Both models correctly identified the core content; minor differences do not affect information retrieval. +- Both models share the same minor errors or are both perfect. +- **Image/Figure processing differs** (one extracts text, one gives bbox, one ignores it), but the main text is accurate. + +【Items to Ignore - Do NOT factor into scoring】 +- Markdown formatting differences (e.g., `# Header` vs `## Header`, `*` vs `-` for lists). +- Layout and typesetting differences (newlines, indentation, alignment). +- Recognition differences in non-body text like Headers, Footers, and Page Numbers. +- Text wrapping and paragraph segmentation nuances. +- Table border styles (e.g., `|---|---|` vs `|:--|--:|`). +- Different but equivalent LaTeX representations for formulas. +- **Image/Figure Processing Differences (ABSOLUTELY IGNORE)**: + - How the model parses image/figure regions is **completely excluded** from the scoring standard. + - Whether it parses as a `figure` field, outputs bbox coordinates, extracts text inside the image, provides a caption, describes the image content, or **completely ignores/skips the image**, these are all considered equivalent. + - Do NOT declare a winner based on image handling. + +【Model 1 Output】: +```markdown +{c1_text} +``` + +【Model 2 Output】: +```markdown +{c2_text} +``` + +【Evaluation Process】 +1. Carefully compare the text content against the original image. +2. Identify errors, omissions, or additions in text recognition for both models. +3. Check the accuracy of table data. +4. Evaluate the correctness, completeness, and semantic equivalence of mathematical formulas. +5. **Ignore image regions**: Confirm that differences in image/figure parsing are not used for scoring. +6. Important: If the substance is the same and only the format differs, judge as a tie. +7. Only declare a winner if there is a significant difference in **content accuracy**. + +【Examples of Ties】 +- Model 1: "# Title", Model 2: "## Title" (Same content, different level). +- Model 1: "* Item", Model 2: "- Item" (Same content, different bullet). +- Formula: Model 1 "$x^2$", Model 2 "$x*x$" (Different LaTeX, same meaning). +- Table data is identical, but column alignment syntax differs. +- Identification is identical, but one model parsed the footer while the other didn't (Judge as Tie). +- **Image handling**: Model 1 outputs an image bbox, Model 2 outputs an image description, Model 3 ignores the image. As long as the main text is accurate, this is a **Tie**. + +【Output Requirement】 Please strictly return the result in the following JSON format: + +{{"winner": "tie", "reason": "Detailed explanation of the judgment, specifically noting the logic for a tie"}} + +The value of "winner" must be one of: +- "1": Model 1 is clearly better in content accuracy. +- "2": Model 2 is clearly better in content accuracy. +- "tie": Both models perform equally in content accuracy (including cases of identical content but different formatting/image handling). + +In the "reason" field, specifically explain: +- If a tie: Explain the consistency of the content and explicitly mention which formatting or image handling differences were ignored. +- If a winner: Specifically point out the accuracy differences (typos, missing words, table/formula errors). +- **Note**: It is better to judge a tie than to incorrectly determine a winner based on minor formatting or image parsing differences. **Content accuracy of the main text is the ONLY standard.** +""" + + return prompt \ No newline at end of file