Databricks Says 300M Chart-JSON Pipeline Tops Four Multimodal Baselines on Answer Accuracy

Databricks says its chart-JSON retrieval pipeline exceeded four multimodal embedding baselines on answer correctness across two chart-focused tests, using a 300-million-parameter text embedding model and three chart images at answer time. The company plans to add the extraction automatically to ai_parse_document, turning chart values into structured text without changing that tool’s interface.

The change targets a familiar retrieval failure. A caption can say what a chart is about, but it may omit the values and labels needed to answer a precise question. Databricks compared two indexes built from the same PDFs: one stored figure captions alone, while the other placed captions and structured chart JSON together in each figure chunk.

Its pipeline uses ai_parse_document to extract document content, ai_prep_search to turn it into retrieval-ready chunks, ai_search to create the index, and Genie for retrieval and answering. The design keeps retrieval text-based: the chart’s extracted values become part of the text representation rather than requiring every indexed page to be embedded as an image.

Structured text did not eliminate the role of visual input. Some questions depend on a figure’s appearance rather than its numeric content, so Databricks also supplied the top three retrieved images to the answering agent. Retrieval did not change, but correctness rose by 4 percentage points on Chart-RAG and 2.6 points on the ViDoRe subset.

Databricks compared its approach with four systems: ColQwen2.5-3B, Qwen3-VL-Embedding-2B, Jina CLIP v2, and CLIP ViT-L/14. It says the chart-JSON setup exceeded all four on answer correctness while passing three images to the answering model. The company characterizes its 300M-parameter text approach as roughly 10 times smaller and simpler than ColQwen2.5-3B’s multi-vector, late-interaction design.

The evaluation covered 310 chart- and infographic-heavy questions from a ViDoRe V3 subset, drawn from a 16,000-page English corpus, and 114 visually grounded questions in Chart-RAG. Databricks built the latter from three chart-heavy reports totaling 378 pages because many ViDoRe chart-focused questions could still be answered from surrounding text.

Answers were graded as correct, partly correct, or incorrect by Gemini-3-Flash against gold answers. Each configuration ran three times and the company reported confidence intervals. On Chart-RAG, Databricks notes that retrieval for the strongest systems was close to saturated because the corpus was small, making answer quality the more meaningful comparison there.

Databricks says chart-JSON enrichment will arrive soon in ai_parse_document and automatically add chart values as structured text. It also says the capability will power Genie One for chart-related questions. The announcement establishes a planned product direction; it does not provide a release date.