Anthralytic’s Substack
Sign in

How Can Humans and AI Improve Each Other’s Interpretations?

Human review is the safeguard every account names and almost none of them specifies.

Anthralytic's avatar
Anthralytic
Nov 13

Part 5 of 6

This is the fifth piece in a six-part series on AI and qualitative interpretation. Part 1 asked when AI strengthens qualitative interpretation and when it reproduces, amplifies, or obscures human biases. Part 2 asked what a model matching human coders actually shows. Part 3 asked when working with AI exposes the assumptions a team shares. Part 4 followed the material from recording to finding. This piece takes up the fourth subquestion: under what conditions can humans and AI identify and correct problems in each other’s interpretations?

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.


Three pieces in this series have ended in the same place. A model does something useful, a model does something wrong, and the thing that catches it is human review. The last piece closed on the harder version of that: if the humans and the model miss the same thing, the review step confirms the omission instead of finding it.

Human review gets named as the safeguard in nearly every practice account I’ve read. It’s rarely specified. Who reviews, against what, with how much time, and with the standing to say the analysis has to change: those details decide whether review is a check or a formality, and they’re usually left out.

This piece asks what review would have to consist of to do the job, in both directions: humans checking AI work, and AI checking human work. Neither has been studied the way it would need to be. The first has been described often and measured rarely. The second has barely been tried.

The most detailed account of human review is also an account of what it costs

In From Algorithms to Evidence, Indran Naidoo, Anoop Sharma, Steven Jonckheere, and Hannah den Boer describe a pilot at IFAD’s Independent Office of Evaluation. Their country case studies run on a structured protocol: interview questions map to sub-questions in the evaluation framework, and every finding has to trace back to a timestamped quote. They built the AI workflow on top of that protocol rather than around it.

The model analyzed each transcript response by response, anchored to the question that prompted it, pulling verbatim quotes with timestamps and the substantive points in each answer. A script mapped the output to anonymized speaker identifiers. Then the model drafted a preliminary synthesis across stakeholder groups. Evaluators did the last stage: correcting misattributions and oversimplifications, then triangulating against portfolio reviews, project documents, and prior evaluations.

The reported gains are substantial. Manual coding of a case study had taken three to four weeks of evaluator time; the piloted workflow processed and mapped transcripts in four or five days. Every response in the transcripts entered the evidence database linked to its quote and timestamp, which they note is more than partial manual coding usually achieves.

The authors are equally direct about what the arrangement cost. The protocol left less room for themes nobody had anticipated; an “unsolicited insights” field caught some extra material, though not all of it proved useful. Breaking transcripts into question-by-question extracts reduced evaluators’ chances to sit with a whole interview and build an intuitive sense of it. And the structured syntheses looked authoritative enough that preliminary outputs risked being accepted without much challenge. Without evaluator validation, they write, the AI narrative risked “flattening the findings into a single message of adequacy.”

This was a pilot alongside work the team had already done manually, so the time comparison is against their own prior experience rather than a controlled trial. What transfers is the shape of the trade-off. The same protocol that made review possible, by tying every point to a quote, is what narrowed what review could notice. An audit trail catches a fabricated quote. It doesn’t catch a question nobody asked.

The other direction has barely been tried

Set against that, consider what a model might catch in human work. It could locate passages that contradict an emerging theme. It could flag a code applied one way in the first week of analysis and another way in the sixth. It could point out that a claim in a draft isn’t supported by the quotes sitting underneath it. It could propose a reading nobody in the room raised.

Each of those is testable, and none requires the model to be right. It only requires the model to be specific enough that a person can check.

One system has built a narrow version of it. Reflexis, which the third piece covered, includes a code drift alert that flags when a code looks like it’s being broadened or misapplied and passes that flag back to the coders. That’s a model catching a human problem, though in a single short session with no comparison condition. Past it I came up close to empty. I couldn’t find studies that hand a model a completed human analysis and ask it to find the problems, and the blind verification study from the second piece is the nearest thing, with a human checking both sides rather than a model checking humans. The field has spent its effort testing AI against human judgment and almost none testing human judgment against anything.

There’s also a reason to be careful about what we’d be measuring. The third piece covered sycophancy, the documented tendency of these models to mirror the user’s framing. A reviewer that folds the moment you push back isn’t much of a reviewer. Whether a model will hold a well-grounded objection under pressure is a property that would have to be measured before anyone leans on AI challenge as a safeguard.

Review is labor, and the labor goes to someone

The last piece used Lea Corsetti, Edwin Asare, and Diana Cekova’s CGIAR chapter for transcription accuracy. A different finding in the same chapter belongs here.

They report that while the baseline tasks sped up, oversight requirements intensified. Review cycles, error correction, and verification became standard parts of the workflow, and they often offset the initial time savings. Overall delivery timelines barely moved. They also describe what they call a technological division of labor: junior staff managed the vast majority of AI operations, verification included.

That last detail is one worth pausing on. The person doing the checking was often the most junior person in the room, checking output that a senior colleague may have already found persuasive. Whether a junior evaluator can say that an authoritative-looking synthesis is wrong, and be heard, is a question about the organization rather than about the model. It’s also a question nobody is studying.

One thing their team did deserves copying: tools that consistently produced inaccurate results were replaced with manual methods. That’s review with teeth, review that can change the workflow and not just the output.

What review needs

Pulling those accounts together, a short list of conditions emerges.

  • Access to the source material, not only the model’s extracts.

  • Independence where the method calls for it, which means forming a judgment before seeing the output you’re checking.

  • A record of reasoning, so a reviewer can see why an interpretation was reached and not only what it concluded.

  • Contextual expertise, because a reviewer who doesn’t know the setting can only check internal consistency.

  • Time, which is the first thing a deadline takes.

  • Authority to revise, because a reviewer who can flag a problem but not fix it has produced a note, not a correction.

Azra Nurkic and Lauren Ropp’s chapter, which the last piece used for documentation bias, contains a concrete version of this. Every AI-generated row in their evidence matrix was marked Accept, Edit, or Discuss. Items marked Discuss went to short sense-making sessions, and the decision was then written back into the codebook and the prompt template, with affected rows corrected. The review changed the instrument, not just the row.

In my own contribution to the book, every AI draft was treated as provisional and went through structured human review, with authorship and version records kept throughout. Beyond checking accuracy, that review covered coherence, whether definitions carried correctly across documents, traceability, and bias. I’ll add the thing I got wrong, since it bears on the conditions: I didn’t seek explicit consent for AI assistance when drafting the terms of reference, and I noticed only partway through the engagement. Consent for AI use has to be asked for repeatedly as the work changes shape. It can’t be assumed from one early conversation.

What research would compare

The useful study here compares arrangements rather than outputs. Four worth running against the same material: human analysis with no AI involvement; a model asked to challenge a completed human analysis; human review of model output; and an iterative exchange where each responds to the other.

What to track is the traffic, not the endpoint. Every proposed correction, who or what proposed it, whether it was accepted or rejected, the stated reason, and what changed in the final interpretation as a result. How long each review took, and who did it.

One caution carries over from the second piece. Agreement after review isn’t evidence that correction worked. A team that accepts every AI challenge and a team that rejects every one both end up agreeing with themselves. The test is whether the revised reading is better supported by the material than the original, which is slower to assess than counting how many flags were raised.

Every account in this series ends with a human checking the work. None of them says how to tell when the checking worked. The final piece brings the five questions together into an agenda for research, and a guide to what’s worth reading.

Leave a comment

Share


Anthralytic builds evaluation tools for organizations that can’t afford an evaluator on staff. Evaluability checks, theory of change builders, and planning assistants, designed so the people doing the work can do their own evaluation, with an evaluator in the loop when it counts.


References

Corsetti, L., Asare, E., & Cekova, D. Experimenting with AI tools in CGIAR Science Group evaluations: A candid assessment of early adoption. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement is not quality: Blind expert verification of human and LLM qualitative coding when human consensus is not ground truth [Preprint]. arXiv. https://arxiv.org/abs/2607.28890

Naidoo, I. A., Sharma, A., Jonckheere, S., & den Boer, H. Generative AI for protocol-driven structured interview analysis in evaluations. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139

Nurkic, A. K., & Ropp, L. Integrating generative AI into thematic analysis: A structured approach for large-scale document review in evaluation. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139

Paschke, S. Threading the needle: Making evaluation affordable and participatory for grassroots nonprofits. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139

Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2023). Towards understanding sycophancy in language models [Preprint]. arXiv. https://arxiv.org/abs/2310.13548

Ye, R., Huang, O., Lee, P. Y. K., Liut, M., Nobre, C., & Kong, H.-K. (2026). Reflexis: Supporting reflexivity and rigor in collaborative qualitative analysis through design for deliberation. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI 2026). https://doi.org/10.1145/3772318.3791275 (open manuscript: https://arxiv.org/abs/2601.15445)

a close up of a person holding a wooden object
Photo by Marco Bianchetti on Unsplash

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.