Adaptive Parsing: Escalating from PyMuPDF to Azure and Vision LLMs

Classic OCR tools like EasyOCR recover words and quietly drop the table structure around them — answers read fine until you check them against the source page. A deterministic check catches obvious failures for free, but a table that parsed into plausible-looking nonsense sails right through it, and the model answers from that nonsense with full confidence. Nothing upstream flagged it, because on paper the parse looked fine. The last line of defence is the LLM reading its own input before the user ever sees the answer.

This article is the second of two parts on adaptive parsing, in Part III of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks: document parsing, question parsing, retrieval, and generation. The first part built the escalation cascade and the deterministic checks that flag a failed parse for free. This part adds the LLM as the last line of defence and walks two real escalations end to end.

where this article sits in the series: Article 10 (adaptive parsing), in Part III — Image by author

📓 The runnable companion walks both escalations end to end: you parse the flat-table page with fitz then re-parse it with Azure, send the figure page to a vision LLM, and watch context_structured flip the generation from a wrong answer to a right one. On GitHub: doc-intel/notebooks-vol1.

The public companion-code repo at doc-intel/notebooks-vol1 — Image by author

PyMuPDF parses a page in five milliseconds for free. A vision LLM on the same page can cost ten thousand times more and take ten seconds. When the question lives in plain prose, the cheap parser wins. When the answer hides inside a table, a figure, a scanned region, or a flattened layout, the cheap parser silently returns nothing useful and the LLM confidently answers the wrong question.

Running the heaviest parser on every page is wasteful. Running the cheapest one everywhere is wrong. The trick is to start cheap and escalate only when something says the cheap parse missed the answer. That signal has to come from the pipeline itself: a chain of checks against the parse output, the question, and the LLM’s own footprint when it tries to use what came out.

The way around this is a feedback loop the pipeline builds for itself. Start with the cheapest parser. Evaluate its output at every check of the pipeline. Escalate to a deeper parser only when at least one check flags the cheap parse as insufficient for the question.

The evaluation is not one check at one place. It is a cascade:

• pre-parsing metadata routes most of the corpus before any extraction
• parsing-time outputs flag flattened tables and opaque figures
• retrieval scoring catches anchors that drift
• generation flags what survived all of the above

Each check is cheaper than the next, more reliable than the next, and asks its own question: “did the parser produce enough to answer?”

The first part caught failures cheaply, before generation. This part is what happens when they slip through: the LLM’s own footprint at generation time (check 7, self-evaluation; check 8, groundedness) flags a parse the deterministic checks missed, and the pipeline escalates the offending page to a deeper parser. Two recurring examples are walked end to end: Table 3 of the Attention paper (Vaswani et al. 2017, under the arXiv non-exclusive distribution license) (page 9, a flat-parsed grid escalated to Azure Layout) and Figure 1 (page 3, an opaque diagram escalated to a vision LLM). Both are real, both come up in enterprise corpora every day.

1. The Generation-Side Checks: Check 7 (Self-Eval) and Check 8 (Groundedness)

Generation is the last stop in the cascade. Its two checks are check 7 (the LLM’s self-evaluation flag, baked into the typed answer) and check 8 (a separate-LLM or NLI groundedness check after the answer is produced). Check 7 is the most expensive and least stable check in the panorama, so this part spends real time on it: two walkthroughs that show it firing usefully (Case A in section 2, Case B in section 3), then a stress test that says the LLM signal alone is not enough (section 4). Check 8 is section 5.

Before the two walkthroughs: most questions do not need any escalation at all. Article 9 ran the standard pipeline on the same Attention paper with the question “What are the options for positional encoding?” and got a clean answer at the first generation pass, context_structured=True, no re-parse needed. PyMuPDF alone was enough because the answer lived in prose. Same for “What are the six CSF Functions?” on the NIST document. The two walkthroughs below are the harder cases, where PyMuPDF could not see what the question needed.

Two questions on the Attention paper, two failure shapes, two parser escalations. The walkthroughs mirror each other on purpose: same data model, different parser at the deep step.

2. Case A: Flat-Table Escalation, End to End

The question: “What is the value of h for the base model in Table of Variations on the Transformer architecture?” The correct answer is 8, readable from the h column of the base row of Table 3 on page 9 of the Attention paper. The cell is filled (not blank) so the answer itself is unambiguous; what stresses the pipeline is the flat-parsed shape of the surrounding grid, which is exactly what the check-2 fingerprint already flagged for free in the first part and what the LLM signal at check 7 also flags below.

2.1. Question, Used Directly

Article 6 builds a question-parsing brick (parse_question) that handles typos, multi-part questions, language detection, and expert-vocabulary expansion. On a question this short and clean, the brick is overkill. We skip it and feed the question text directly to retrieval and generation. The pipeline does not break; we just choose not to call one of the four bricks because the input is already in the shape downstream wants.

The keywords we pick by eye for the retrieval call below: just the table-caption string, “Table 3: Variations on the Transformer architecture”. One specific phrase is enough on this corpus.

2.2. Parsing: PyMuPDF, with Bounding Boxes

The standard pipeline starts cheap with PyMuPDF. One function, one line:

line_df = fitz_pdf_to_line_df("data/paper/1706.03762v7.pdf")
page_df = build_page_df(line_df)

# line_df.shape ~ (1500, 8)
# columns : page_num, line_num, text, x0, y0, x1, y1, character_count

The first eight lines of the captured line_df (page 1, before we ever look at page 9):

First eight line_df rows; every line carries its bbox in PDF points — Image by author

2.3. Retrieval: Which Method Finds Page 9, and How

Article 7 covers four retrieval methods: keyword matching, embedding similarity, TOC matching, and an LLM-orchestrator on top. For this question, keyword retrieval is sufficient because the question mentions a specific string (“Table 3”) that the table caption echoes verbatim on page 9. We pass that exact string as the only keyword:

from docintel.retrieval import retrieve_pages

retrieved_pages_df, filtered_line_df = retrieve_pages(
    page_df, line_df,
    keywords=["Table 3: Variations on the Transformer architecture"],
    top_k=5,
)

The captured retrieved_pages_df:

One page returned: the caption phrase is unique to page 9 — Image by author

The filtered_line_df returned alongside retrieved_pages_df carries every line of page 9, ready for the generation brick.

2.4. First Generation: gpt-4.1 on PyMuPDF, the LLM Flags the Parse

The generation brick (llm_answer_with_evidence, Article 8) takes the filtered lines and the question, calls gpt-4.1 with a Pydantic schema as the output contract. The AnswerWithEvidence schema already carries context_structured (added in Article 8), so the orchestrator has a one-bit trigger to read on every answer:

from docintel.generation import llm_answer_with_evidence
from docintel.generation.qa.schemas import AnswerWithEvidence

pass1: AnswerWithEvidence = llm_answer_with_evidence(
    question=("What is the value of h for the base model "
              "in Table of Variations on the Transformer architecture?"),
    filtered_line_df=fitz_p9,    # PyMuPDF lines for page 9
)
# AnswerWithEvidence carries :
#   answer, start/end_page_num, start/end_line_num, confidence,
#   justification, caveats,
#   context_structured  <-- the binary trigger for the cascade
#   complete_answer_found, llm_discovered_keywords

The captured response, copy-pasted from the real run:

Pass 1: correct answer "8", conf 0.99, but context_structured=False flags the parse — Image by author

{
  "answer": "8",
  "start_page_num": 9,
  "start_line_num": 25,
  "end_page_num": 9,
  "end_line_num": 25,
  "confidence": 0.99,
  "justification": "In the table listing parameters for different Transformer ...",
  "caveats": [],
  "complete_answer_found": true,
  "context_structured": false
}

The answer is correct and the confidence is high, but context_structured=False signals that the LLM found the value despite the parse quality, not because of it. That flag is the trigger: the orchestrator escalates page 9 to Azure Layout for a structured re-parse, which recovers the full grid with row and column headers intact — giving the pipeline a verified, structured context rather than one the model had to reconstruct from a flattened stream of tokens.