Anthralytic’s Substack
Sign in

When does AI strengthen qualitative interpretation, and when does it reproduce, amplify, or obscure human biases?

Part 1 of 6

Anthralytic's avatar
Anthralytic
Nov 9

This is the opening piece in a six-part series on AI and qualitative interpretation. Researchers bring contextual knowledge, theoretical commitments, and assumptions to qualitative work. AI models bring influences of their own, and they also reflect the frameworks and material researchers give them. The series asks how those influences interact: when working with AI helps researchers notice what they overlooked, and when it reinforces or hides problems they already had. This piece sets out the central question. The next four each take up one part of it, and the last turns the five questions into an agenda for research.

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.


Using GenAI in qualitative work is no longer a fringe experiment. Evaluators are using it to review documents, suggest codes, and draft themes. A 2026 review of health research alone found 42 published studies doing it, most of them from 2025.

In From Algorithms to Evidence, a new collection of evaluators’ accounts of working with AI, fellow contributor Geordie Fung describes a problem: the model mistook how often something was said for how much it mattered. His case is the design of a monitoring, evaluation, and learning framework for a policy think tank. One stage of that work used GPT-4 to identify the dominant themes across more than 100 of the think tank’s past outputs. Regional partnerships surfaced repeatedly, so the model coded them as central. Budgetary trade-offs and policy coordination failures appeared less often, so they were under-emphasized, even though they carried more strategic weight.

It’s fair to ask how the model could have known otherwise. Nothing in the documents marked budgetary trade-offs as more important than partnerships. That judgment came from the evaluators’ understanding of the organization and its strategy. Fung doesn’t say whether any of that context was given to the model, so I can’t tell you whether it ignored what it was told or was never told. Either way, the evaluators had to step in and judge whether all those references to partnerships reflected real contribution or “merely reflected organisational habits.”

The same honesty applies to the evaluators. Fung reports their judgment that the model had it wrong. But how do we know the evaluators were right? The chapter doesn’t show how their judgment was tested: whether anyone went back to the documents, checked it against the think tank’s actual strategy, or asked staff. I have no reason to doubt the correction. It looks right because it matches what experienced evaluators would expect, and whether that is a good enough standard is the question this series is asking.

The habit was the organization’s, not the model’s

The frequency problem is easy to blame on the model, but the emphasis on partnerships came from the think tank’s own publications. The model didn’t invent it; it counted it. What looked like a model error was partly an institutional habit, handed back as an analytical finding.

In Fung’s account, the evaluators caught the problem because the output clashed with their own sense of what mattered. That raises a harder question. If the evaluators had shared the organization’s habits, the same output would have looked accurate, and there would have been nothing to catch. A model that reproduces a team’s assumptions confirms them, and nobody in the water notices the water. I’m not claiming that happened in Fung’s case. I’m saying his case shows how it could happen, and that we don’t yet know how often it does.

The same tool could show us what we missed

The reverse is also possible: a model could expose assumptions a team shares. A model doesn’t get tired at the fortieth transcript, and it has no stake in the story the team is starting to tell. It might locate passages that contradict an emerging theme, propose a defensible alternative grouping, or flag a code applied unevenly across interviews. Any of those could strengthen an analysis, including a human one.

Whether that counts as help can’t be settled by how useful it feels. Three things would need to be true: the challenge is grounded in the material, it changes the interpretation, and the revised reading is better supported than the original, not just different. Those conditions can be tested, which is one reason this is a research question rather than a matter of opinion.

Qualitative practice already has ways to catch this

Qualitative researchers haven’t been waiting for AI to worry about shared assumptions. Team-based approaches use independent coding, disagreement, and codebook revision for exactly this purpose. MacQueen and colleagues describe revising categories that had invited unsupported assumptions about participants’ socioeconomic circumstances. When calibration works, it surfaces conceptual problems. It isn’t just training people to agree.

What calibration looks like when one of the coders is a model is a different matter, and almost nobody has studied it. A model doesn’t disagree the way a colleague does. It will usually produce a code when asked, revise when pushed, and rarely hold its ground. Whether a team can calibrate with something that accommodating, or whether it quietly becomes the team’s common denominator, is an open question. I’ll come back to the little evidence there is at the end of this piece, and to the question itself in the third.

Even the advocates of these methods acknowledge their limit. O’Connor and Joffe, who argue for appropriate use of intercoder reliability, note that agreement may sometimes reflect what they call interpretive convergence: coders coming to read the data the same way rather than reading it well. Good practice reduces that risk. It can’t remove it, and adding a model to the team doesn’t change that on its own.

Subjectivity isn’t the problem

Before going further, I want to be precise about the word bias, because the whole series depends on it. Not every interpretive difference is a flaw. Contextual knowledge, theoretical commitments, and lived experience often deepen interpretation. In reflexive thematic analysis, Braun and Clarke treat researcher subjectivity as an analytic resource rather than a threat to be contained. Two coders who read a passage differently haven’t necessarily made an error. One of them may simply have noticed more.

When I say bias in this series, I mean something narrower: assumptions the evidence doesn’t support, systematic discounting of certain accounts, consequential omissions, or failure to engage contrary evidence. This matters for how we judge AI, because the most common test, agreement with human coders, can’t detect most of these. Agreement shows whether people applied a framework consistently. It can’t show whether the framework was the right one, or whether everyone missed the same thing.

The research complicates the benchmark

Recent studies take up parts of this question. Read together, they show that comparing AI to human coders answers less than it seems to.

Ashwin, Chhabra, and Rao show the risk is real. They compared LLM coding of interviews with Rohingya refugees and their Bangladeshi host community in Cox’s Bazar against careful human annotations. The models’ errors weren’t random with respect to who was speaking: for one model, errors tracked household characteristics such as refugee status, education, and income on 10 of 19 codes. Errors that fall unevenly across groups can distort what a study concludes about those groups. That’s a concrete risk, not a vague worry about AI bias. It’s also measured against human codes, which can’t by themselves show how biased the humans were.

Two other studies question the yardstick itself. Parfenova and colleagues found that the labels closest to a reference set weren’t always the ones experts preferred. A preprint by Liu and colleagues pushes further. Human-model agreement trailed human-human agreement, which would normally count against the model. Yet when a blind expert verifier compared code sets without knowing their source, the preferences split almost evenly between human and model, and for some codes the authors report that human consensus reflected a shared bias, with the verifier favoring the model’s reading. That’s one verifier in one setting, and a statistical tie isn’t proof of equivalence. What it does is put human consensus under the same scrutiny as the model.

A preprint from Marston and colleagues shows what averages can hide. They benchmarked 46 models on 150 synthetic humanitarian transcripts. Several matched experienced human coders on deductive coding overall, yet struggled with indirectly stated needs and with protection concerns such as physical safety and discrimination. In humanitarian work those are among the accounts that matter most, so strong aggregate performance can sit beside the most consequential omissions. Synthetic transcripts leave open how this holds up in the field.

Yesterday’s results are a starting point

None of this settles into a verdict, partly because the systems keep changing. Fung worked with GPT-4 about a year ago. Newer models may not confuse frequency with significance the same way, or they may do it more persuasively. Neither should be assumed, which is why earlier cases are worth repeating with current systems rather than filing as settled.

The task matters too. Extracting passages, applying an existing code, building categories, constructing themes, and producing explanations are different kinds of work, and a model can be dependable at one and unreliable at another. Good performance on one says little about the others.

The interaction is the thing to study

Here is the gap I think matters most. Nearly all the evidence so far compares finished outputs against a reference: human codes on one side, model codes on the other. But practice doesn’t happen that way. It happens in the exchange, where a model’s suggestion meets a researcher’s assumption and one of them gives way. Whether that exchange sharpens interpretation or quietly narrows it is a question none of these studies was designed to answer. Studies of the exchange itself exist, but they’re early and small. CoAIcoder had 32 pairs of novice coders work with and without a shared AI suggestion model, and found that sharing the model sped up early agreement while appearing to reduce the variety of codes the pairs produced. The model was a pre-generative classifier trained on the coders’ own history, so this says little about today’s chatbots. Reflexis, a tool designed to prompt reflection and structure disagreement in reflexive thematic analysis, was tried by 12 researchers in a single session, with no comparison group. Both show the question can be studied. Neither comes close to answering it.

The next four pieces take the question apart: what should count as stronger interpretation and who gets to judge; when working with AI exposes the assumptions a team shares and when it reinforces them; whose meanings survive as accounts become codes, themes, and findings; and how humans and AI might check and correct each other’s interpretations, in both directions.

A model that agrees with us is the one we’re least equipped to check.


Anthralytic makes evaluation accessible to resource-constrained organizations through AI-assisted tools and hands-on consulting, with an evaluator in the loop.

Share

Leave a comment

User's avatar
Join Anthralytic’s subscriber chat
Available in the Substack app and on web


References

Ashwin, J., Chhabra, A., & Rao, V. (2025). Using large language models for qualitative analysis can introduce serious bias. Sociological Methods & Research. https://doi.org/10.1177/00491241251338246 (open access)

Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. https://doi.org/10.1080/14780887.2020.1769238 (author copy: https://uwe-repository.worktribe.com/output/6631950)

Fung, G. Designing with generative AI: Using GPT-4 to build a monitoring, evaluation, and learning framework. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139

Gao, J., Choo, K. T. W., Cao, J., Lee, R. K.-W., & Perrault, S. (2024). CoAIcoder: Examining the effectiveness of AI-assisted human-to-human collaboration in qualitative analysis. ACM Transactions on Computer-Human Interaction, 31(1), Article 6. https://doi.org/10.1145/3617362 (open preprint: https://arxiv.org/abs/2304.05560)

Gierbolini-Rivera, R. D., Franco Silva, M., de Paula da Silva, A. A., Brownson, R. C., Parra, D. C., Kepper, M. M., & Eyler, A. A. (2026). Generative artificial intelligence for qualitative methods in health research: Rapid review. Journal of Medical Internet Research, 28, e98551. https://doi.org/10.2196/98551

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement is not quality: Blind expert verification of human and LLM qualitative coding when human consensus is not ground truth [Preprint]. arXiv. https://arxiv.org/abs/2607.28890

MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998). Codebook development for team-based qualitative analysis. Cultural Anthropology Methods, 10(2), 31–36. https://doi.org/10.1177/1525822X980100020301

Marston, J., Kreutzer, T., Garnier, S., Boone, E., Pham, P. N., & Vinck, P. (2026). Can large language models reliably code qualitative humanitarian data? A benchmark study against human expert adjudication [Preprint]. arXiv. https://arxiv.org/abs/2606.26541

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19. https://doi.org/10.1177/1609406919899220

Parfenova, A., Marfurt, A., Pfeffer, J., & Denzler, A. (2025). Text annotation via inductive coding: Comparing human experts to LLMs in qualitative data analysis. In Findings of the Association for Computational Linguistics: NAACL 2025 (pp. 6471–6484). https://aclanthology.org/2025.findings-naacl.361/

Ye, R., Huang, O., Lee, P. Y. K., Liut, M., Nobre, C., & Kong, H.-K. (2026). Reflexis: Supporting reflexivity and rigor in collaborative qualitative analysis through design for deliberation. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI 2026). https://doi.org/10.1145/3772318.3791275 (open manuscript: https://arxiv.org/abs/2601.15445)

Team collaborating with sticky notes on glass wall
Photo by Vitaly Gariev on Unsplash

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.