Original title: KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records
Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim, S. M. Shah Nawaz Hossain +2 · September 2026
In Bangladesh, official land records called RS Khatians use a rare base-16 fraction system known as Ana-Ganda-Kora-Kranti-Til. These handwritten records have unique symbols, lack standard computer fonts, and are missing from optical character recognition pipelines and text tokenizers. Because millions of land titles rely on these documents, researchers created KhatianDoc to test whether modern vision-language artificial intelligence models can read them. The researchers gathered 107 real land records and built four tasks: recognizing symbols, converting base-16 fractions to decimals, extracting form fields, and answering legal questions. They tested six multimodal models in a zero-shot setup. The models failed completely. Across five question categories, covering 39.3% of the stratified set, every single model scored zero. On the math conversion task, models performed worse than just guessing a constant mean, showing completely uncorrelated answers rather than close guesses. The benchmark demonstrates that these models entirely lack the capability to process these legal land records.
Imagine you receive a handwritten letter from a relative overseas, but it's written in a shorthand you've never seen and uses a currency you don't recognize. You might guess at some words, but you can't understand the amounts or the full message. In this paper, the land records are the letter, the base-16 fractions are the unfamiliar shorthand and currency, and the AI models are the reader unable to decode it.
Land records in Bangladesh are handwritten and use a unique number system, making them unreadable to current AI. This affects millions of people in land disputes, as automated tools cannot verify these authoritative documents. It shows a critical gap in AI for legal and cultural applications.
The abstract does not specify limitations of the benchmark's generalizability, but it states that the work diagnoses an absence of capability rather than a performance gap. It corrects scoring artifacts and flags metric inflation, suggesting that some results should be interpreted carefully.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records — Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.