Perovskite Literature AI Assistants Are Only "Half Right": The Real Weakness Is Not the Model but the Data Table
In brief
A new arXiv preprint uses 755 quantitative questions and 45,441 measured device records to score "perovskite literature AI assistants": zero-shot answers reach only 1.6–6.5%, conventional passage retrieval at most 16–23%, and only when the model queries a structured device table with read-only SQL does accuracy rise to about 51%. We briefly review the work and discuss what it means for PV R&D and technology-transfer engineers using AI for literature search, and for companies building internal experimental databases.
Over the past two years, many R&D colleagues have got used to handing papers to AI and asking: "What is the highest perovskite device efficiency reported in the last two years?" or "What is the average open-circuit voltage of devices using a certain hole transport layer?" The AI answers fluently and confidently, but is it right? A preprint posted on arXiv a few days ago scores this kind of question for the first time against reference answers computed from measured data. The result: even the best approach gets only half right.
01What the paper does
The paper, "Benchmark for Structured Retrieval-Augmented Generation in Perovskite Photovoltaics" by Qiwei Han, Chi Zhou and Gaoxiang Yuan, was posted on arXiv in late October (arXiv:2610.25575) and runs to eight pages.
Its starting point is simple. Most benchmarks for "literature AI assistants" judge how similar an answer is to a reference text, or have another large model grade it; few check answers against real experimental data. Yet in experimental materials fields like perovskites, the questions researchers care most about are often quantitative and can only be answered by aggregating across dozens or hundreds of papers.
- Question set: 755 quantitative questions, each with a reference answer computed directly from a device database rather than written by people.
- Ground truth: 45,441 measured device records, of which 40,889 are manually curated records up to 2021 and 4,552 are records extracted from new literature from 2022–2026.
- Text corpus: 25,662 papers and 158,274 passages.
- Hardware: a 14B-parameter open-source model on a single 24 GB GPU, reproducible on an ordinary workstation.
Questions are split into "historical literature" (2021 and earlier) and "new literature" (2022–2026), the latter testing how the model handles new knowledge beyond its training data.
02The scorecard: from almost all wrong to half right
The authors compared six retrieval architectures in total. One table tells the story:
| Approach | Method | Accuracy |
|---|---|---|
| Zero-shot | No documents; the model answers "from memory" | 1.6%–6.5% |
| Passage retrieval (RAG) | Relevant passages from 158k text passages fed to the model | At most 16%–23% |
| Structured query (SQL) | The model writes read-only SQL to query the device table | 51.5% (historical) / 50.8% (new literature) |
Table 1 Accuracy of different retrieval approaches (source: arXiv:2610.25575 abstract)
Figure 1 Accuracy of different approaches on 755 quantitative questions (source: arXiv:2610.25575)
Three numbers are worth dwelling on. First, zero-shot accuracy is in single digits, showing that the perovskite data a large model has "memorised" is essentially unusable; asked for specific values, it is mostly making them up. Second, adding passage retrieval lifts accuracy to around 20%, but it quickly plateaus. Third, switching to table queries jumps straight to around 50%, and the historical and new-literature groups score almost the same, showing that the gain comes not from what the model has "seen" but from the data itself.
The abstract states that the structured approach has an advantage on the order of thirty percentage points over "hybrid text retrieval" (the arXiv page displays the figure as "33.01.8", apparently a formatting glitch; refer to the PDF for the exact value).
With the same 14B model, simply changing how data is "fed" lifts accuracy from about 20% to about 50%: the ceiling is set by how the data is organised, not by model size.
03Why does passage retrieval fail?
The key is the question type. The paper points out that "aggregation questions" involving extremes, distributions and conditional counts are almost unsolvable with text retrieval alone.
Take a familiar example: "How many inverted perovskite cells reported since 2023 have PCE above 25%?" Answering it means finding every qualifying device across thousands of papers, comparing them one by one and counting. Passage retrieval works by "finding the dozen or so most relevant passages", so it can only ever see the tip of the iceberg, like flipping through ten pages of a lab notebook and being asked for the whole lab's yield distribution over three years.
A structured table turns this into a single line of SQL: filter by architecture, year and efficiency, then count. The model only has to "translate" natural language into a query; the arithmetic is left to the database, with no need to "remember" or "guess".
Figure 2 Passage retrieval vs structured query (schematic)
- Text retrieval is good at: definitions, mechanisms and the specific methods of a given paper, answering "what" and "why".
- Structured queries are good at: maxima, averages, distributions, counts and trends, answering "how many" and "which is best".
- Combining the two: use the table to find target devices, then go back to the original papers for process details; that is the complete R&D query chain.
04Three lessons for PV R&D
Although this is a perovskite paper, its conclusions apply across the PV materials field.
- Draw a line for AI literature search: for mechanisms and ideas, AI is a good helper; for specific values and statistical conclusions, always check whether it gives a traceable data source. Even the best approach in this paper still gets about half the questions wrong.
- For company databases, "table" before "model": many companies plan to build internal knowledge bases with large models by feeding in years of experiment reports, test reports and failure analysis reports. The signal from this paper is that unless key parameters such as material batches, process windows, IV data and reliability degradation are first systematically extracted into structured tables, the AI, however clever, can only "try its luck" among passages. The authors conclude exactly this: systematic table extraction is a prerequisite for literature-scale quantitative queries.
- Data standards are worth more than model choice: how fields are defined, how units are unified and how test conditions are recorded, this "dirty work" determines how far AI can go. For materials introduction, reliability and certification teams, standardising experimental data records now lays the foundation for the AI assistants of a few years from now.
05In the context of similar work
Placed among perovskite AI work of the past two years, the paper's position becomes clearer:
- Data foundation: the Perovskite Database Project (Jacobsson et al., Nature Energy 2022) manually curated data on more than 42,400 devices from over 15,000 papers, with up to about 100 parameters per device, covering literature up to February 2020. The 40,000-plus manually curated records "up to 2021" in this paper are of similar scale and may well extend that database, but this needs confirming from the full text.
- Text route: Perovskite-LLM, published at Findings of EMNLP 2025 by a team at the Hong Kong University of Science and Technology (Guangzhou), built a knowledge graph from 1,517 papers and more than 55,000 question-answer pairs, and fine-tuned 8B and 7B models; its evaluation relies mainly on Rouge-L and "LLM-as-judge" scoring. This is exactly the "similarity and AI grading" style of evaluation the new paper criticises, and the two are worth reading side by side.
- Table extraction: a 2025 Scientific Data paper by a City University of Hong Kong team used rule-based natural language processing tools (ChemDataExtractor and others) to automatically extract 30 fabrication and device fields from 3,164 papers, with an average automated-validation accuracy of about 0.90. This shows that the step of "turning literature into tables" itself carries an error of roughly 10%, which propagates directly into downstream question-answering accuracy.
In other words, the field is moving from "getting models to understand the literature" to "first turning the literature into queryable data", and extraction quality will become the next bottleneck.
06Our take
What deserves credit: computing ground truth from measured data rather than "having AI grade AI" is a solid evaluation approach, and running experiments with a 14B open-source model on a single GPU means SMEs and university groups can readily learn from and reproduce the work.
There are also several caveats:
- Preprint status: the paper has not yet been peer-reviewed, and its conclusions await formal publication and peer scrutiny.
- Details to confirm: the abstract does not state the name of the 14B model, the source of the 40,000-plus manually curated pre-2021 records, or whether code and data are open.
- Where the new data comes from: the 4,552 records for 2022–2026 were "extracted", so the extraction method and quality themselves affect the reliability of the ground truth; given extraction accuracy of about 90% in comparable work, this part of the ground truth should be treated with caution.
- Half is still not enough: for R&D decisions, getting half wrong means every conclusion still needs manual checking. Whether the remaining errors come from wrong SQL, missing table fields or errors in the data itself is worth looking for in the full paper's error analysis.
07Final thoughts
When we talk about AI in materials R&D, attention often goes to "which large model is stronger". This paper reminds us that for quantitative questions, AI's ceiling is often set by the data table nobody wants to compile.
For PV professionals, this is actually good news: models can be bought and swapped, but well-structured experimental data with consistent definitions can only be accumulated by teams that dig deep over the long term, and that is the real moat. We will keep following the formal version of this paper and real-world cases of AI in PV materials R&D and reliability analysis. What problems has your team met when using AI for literature search or building experimental databases? We would be glad to hear from you.
AI for literature search: trust it on "why"; on "how many", first ask where the data table is.
References
- Qiwei Han, Chi Zhou, Gaoxiang Yuan. Benchmark for Structured Retrieval-Augmented Generation in Perovskite Photovoltaics. arXiv:2610.25575 [cs.LG], 2026.
- T. J. Jacobsson, et al. An open-access database and analysis tool for perovskite solar cells based on the FAIR data principles. Nature Energy, 2022, 7: 107–115.
- Xiang Liu, et al. Perovskite-LLM: Knowledge-Enhanced Large Language Models for Perovskite Solar Cell Research. Findings of EMNLP 2025.
- A. Valencia, et al. Auto-generating a database on the fabrication details of perovskite solar devices. Scientific Data, 2025, 12: 270.
A PV R&D and technology team with 10+ years of experience | In-depth technical knowledge sharing
Materials · Reliability · Failure Analysis · Standards & Certification · Products · Processes
Exploring AI + PV; collaboration welcome
Disclaimer:
1. Original content reflects the authors' personal views only, is for reference only and does not constitute investment advice.
2. Reposted and cited content is credited to its source, out of respect for original work.
3. Some content was produced with AI assistance (chart generation, data retrieval, concept explanations, etc.); the core framework and key content were written by our technical team and reviewed by people.
4. If anything is inaccurate or infringes your rights, please leave a message or contact us and we will deal with it.