Anthralytic’s Substack
Sign in

What Have We Learned When a Model Matches the Humans?

Matching human coders isn’t the same as getting it right.

Anthralytic's avatar
Anthralytic
Nov 10

Part 2 of 6

This is the second piece in a six-part series on AI and qualitative interpretation. Part 1 asked when AI strengthens qualitative interpretation and when it reproduces, amplifies, or obscures human biases. This piece takes up the first of four subquestions: what should count as stronger interpretation, and who gets to judge? It starts where most of the evidence starts, with a model matching human coders, and asks what that match actually shows.

Thanks for reading Anthralytic’s Substack! Subscribe for free to receive new posts and support my work.


For readers outside qualitative research, coding means reading through interviews or documents and tagging passages with short labels, called codes, that name what each passage is about, so that patterns can be found across the whole set. Research teams have long checked this work by having two or more people code the same material independently, then comparing their results and measuring how often they agree. The most common evidence that AI can code borrows that practice: a team codes a set of transcripts, a model codes the same set, and the model’s agreement with the humans is measured the way one human’s agreement with another would be. When the model agrees with the humans most of the time, the result is reported as reassurance: the model performs about as well as a person.

I want to take that result seriously and then ask what it establishes. My argument is that it establishes something real but narrow. It shows the model can apply a framework the way a particular group of people applied it. It doesn’t show the framework was the right one, that the people applied it well, or that the finished analysis says anything true about the people whose words were coded. Those are different questions, and this piece is about keeping them apart.

Agreement measures consistency, and sometimes that’s enough

Team-based coding works through written definitions for each code, independent coding, comparison, discussion, and revision, with disagreements treated as useful because they show where a definition is vague or where two coders understand a concept differently. That process is what gives agreement its meaning. When a team applies a codebook across hundreds of transcripts, the codes have to mean the same thing to everyone, or the counts and comparisons that follow are meaningless. O’Connor and Joffe argue that intercoder reliability earns its place for exactly these reasons: it keeps the work systematic, it forces the team to talk, and it gives readers a reason to trust the coding.

So when a model matches human reference codes, that is evidence of something. It can apply the codebook the way the team did. For a large study with a settled framework, that’s worth knowing.

The clearest case I know of comes from the book. In From Algorithms to Evidence, Steffen Bohni Nielsen and colleagues describe labelling about 18,000 evaluation publications by domain, such as education, public health, or international development, using twelve categories. A senior evaluator worked iteratively with ChatGPT-4o, feeding it batches of training data and checking its labels. ChatGPT alone never reached the accuracy they wanted. The team then had it build a random forest model, a conventional machine learning method, and ran that in R and Python over the full set. On the roughly 1,000 publications the evaluator had labelled by hand, the final model matched those labels 99.5 percent of the time; the chapter doesn’t say whether that was measured on publications the model had already learned from, which would make the number look better than it is. The other 17,000 were labelled by the model alone. The work took about twelve hours instead of the 150 or more that manual labelling would have needed.

The authors describe the task as relatively simple and the model as a work horse, and that framing is what makes the result instructive. Agreement with one experienced coder was the right test here because of what the task looked like: twelve fixed categories, applied to publication metadata, with a target accuracy set in advance. The reference labels were the senior evaluator’s, and for a task this clear there was no reason to doubt them. (The headline number belongs to the random forest model, not to ChatGPT.) Coding an interview about someone’s experience of losing their housing differs on every one of those counts, and it’s worth saying why. The categories often aren’t fixed in advance; they take shape as the analyst reads. The information isn’t sitting in a title or an abstract; it’s in what the person said, how they said it, and sometimes what they didn’t say. And two experienced analysts can read the same passage differently without either being wrong, so there’s no single right label to measure against. The phrase “AI can code qualitative data” covers both tasks. Nielsen and his colleagues were careful to say which one they were doing. The risk is in how a result like theirs travels: a number earned on a bounded labelling task gets repeated as evidence about interpretive work the authors never claimed to be doing.

Applying codes is one stage of the analysis, not the whole of it

Qualitative analysis is several different kinds of work. Agreement is a test of one of them, and it helps to name the others. Someone decides what to study and which concepts matter. Someone decides what the codes are and where their boundaries fall, whether that happens in advance from a framework, emerges from the data as the reading goes, or, as in most projects, both: code a batch, find the definitions don’t fit, revise, recode. Someone applies the codes. Then someone groups them into themes, decides what the themes mean, and writes an account for readers. Agreement tests whether the codes were applied consistently. It says nothing about the judgments that shaped the codes or the ones that follow from them, even when those judgments were being made in the same sitting.

O’Connor and Joffe make this point themselves. Intercoder reliability, they write, is “no guarantee of the trustworthiness” of the data collection that came before it or the theme development and reporting that come after. A codebook can be applied with perfect consistency and still miss what matters in the transcripts. If it does, a model that matches the reference codes will miss the same thing, and the match will look like success.

The reference itself is someone’s judgment

There’s a further problem with agreement, and it’s the one that gets at the second half of this piece’s question: who judges. Agreement is always agreement with some reference, and the reference was made by people. Two recent studies put that reference under scrutiny. In both, the coders who made the reference, the experts who judged the results, and the models came apart from each other, and there was no neutral place to stand to say which of them was right.

Even within a codebook approach, agreement with the reference and quality as judged by experts can come apart. Parfenova and colleagues tested six open-source language models on inductive coding, where the task is to write a short label capturing the main idea of a text segment. The reference labels came from the consensus of three to five human coders. Three expert qualitative researchers then coded a small subset themselves, and the human and model labels were rated by experts.

The two yardsticks disagreed. The experts’ own labels often differed from the reference but were, in the authors’ words, “often rated more favorably by other humans.” Some models matched the reference more closely and were rated lower. Humans did better on difficult segments, models on easy ones.

Here is what I take from this. The reference labels record what one group of coders agreed on earlier. The expert ratings record what qualified people preferred when they saw the labels side by side. Those are different measures, and neither is the truth about the text. A model that scores well on the first can still produce labels an expert would reject, and the other way around. The study is small, 15 segments and three experts, so treat it as a demonstration that the gap exists rather than a measure of its size.

A 2026 preprint by Liu and colleagues pushes the same point further. It’s the study in this series that most directly questions the human benchmark. Three trained coders and five language models applied a 72-code codebook to 2,560 messages that teachers had sent to an AI platform. The humans had been through three rounds of calibration. On the usual measure, the models looked worse: agreement between humans was 0.52 on the Jaccard index, agreement between humans and models was 0.30.

Then the authors did something unusual. They gave 855 pairs of code sets to an independent expert, a researcher with a doctorate in education who hadn’t taken part in the coding and didn’t know which source produced which set. The expert preferred the human coding 51.5 percent of the time and the model coding 48.5 percent, a difference that isn’t statistically significant. On several codes, the humans had agreed with each other on a narrow application that the expert judged as missing relevant material. The authors conclude that for those codes “human consensus encoded shared bias that the verifier rejected.”

The lesson from this study is narrow but important: when human coders agree with each other, they may be agreeing on a mistake, and the usual agreement metrics won’t catch it. What the study doesn’t show is that the models coded as well as the humans. A difference that isn’t statistically significant isn’t evidence that two things are equal. There was one verifier, and the authors say a different expert might have sided with the humans. And the verifier is also a human, so the question the study asks about the coders (are they right just because they agree?) applies to the verifier too (is she right just because she’s the outsider?). Blind review adds a second opinion. It doesn’t settle whose opinion is correct.

Which test fits depends on the tradition

Reflexive thematic analysis came up in the first piece, where I used Braun and Clarke to separate subjectivity from bias: in that approach, the researcher’s perspective is part of the method, and two analysts reading the same passage differently haven’t made an error. That distinction matters again here, because it changes what agreement can tell us. There is no fixed codebook in reflexive analysis; codes are written, merged, and renamed throughout, so there is nothing stable to measure agreement against. If disagreement isn’t error in that tradition, agreement can’t be a measure of quality in it either. Braun and Clarke put it bluntly, calling demonstrations of coding reliability in reflexive analysis “illogical, incoherent and ultimately meaningless.” Not everyone accepts that. Researchers in codebook traditions, O’Connor and Joffe among them, argue that reliability checks have a place in thematic analysis too, and in applied and evaluation work their approach is at least as common. The point for this series isn’t who is right. It’s that a serious body of qualitative researchers rejects agreement as a quality measure, so a study that reports agreement as evidence of quality has already taken a side, perhaps without saying so.

What a fairer test would look like

If agreement answers a narrow question, what would answer the broader one? I’d start by matching the test to the task and the tradition. For applying an agreed codebook, agreement is fine, with attention to where the disagreements cluster. Parfenova’s difficulty finding suggests they won’t be random.

For developing categories, building themes, and interpreting what they mean, agreement doesn’t help, and other checks do. What follows is, in effect, a rubric: a set of questions to ask of any analysis, human or AI-assisted, with the caveat that answering them well is evidence of quality, not a certificate of it. Can each claim in the analysis be traced to passages that support it, and do the passages actually support it when you read them? Does the interpretation hold up against what’s known about the setting? Did the analysis find the passages that cut against its themes and deal with them, or did it skip them? Where it’s appropriate, do the people whose accounts were analyzed recognize the result? That last check needs care. Participants can endorse a flattering account, and no single participant can see the patterns across everyone else’s. Their recognition is evidence, not a stamp of validity. Finally, when two analyses of the same material reach different explanations, which explanation accounts for more of what’s there?

These checks are slower and harder than calculating agreement, which is why agreement gets used. They’re also the checks that would tell us whether an AI-assisted analysis was better or only more consistent.

Which brings back the question of who judges. The studies above offer three kinds of judge, the original coders, outside experts, and the participants, and each has limits the others don’t fix. The honest answer is that no single judge settles it, and a study claiming improvement should say which judges it used and what each could and couldn’t see.

A model that matches our codes has learned how we read. Whether we read well is a question the match can’t answer, and it’s the question the next piece takes up: when working with AI exposes the assumptions a team shares, and when it reinforces them.


Anthralytic builds evaluation tools for organizations that can’t afford an evaluator on staff. Evaluability checks, theory of change builders, and planning assistants, designed so the people doing the work can do their own evaluation, with an evaluator in the loop when it counts.

Thanks for reading Anthralytic’s Substack! This post is public so feel free to share it.

Share

Leave a comment

User's avatar
Join Anthralytic’s subscriber chat
Available in the Substack app and on web

References

Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. https://doi.org/10.1080/14780887.2020.1769238 (author copy: https://uwe-repository.worktribe.com/output/6631950)

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement is not quality: Blind expert verification of human and LLM qualitative coding when human consensus is not ground truth [Preprint]. arXiv. https://arxiv.org/abs/2607.28890

MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998). Codebook development for team-based qualitative analysis. Cultural Anthropology Methods, 10(2), 31–36. https://doi.org/10.1177/1525822X980100020301

Nielsen, S. B., Sutter, A., Bourgeois, I., Montrosse-Moorhead, B., & Lemire, S. How to train a work horse: Using generative AI for labelling data. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19. https://doi.org/10.1177/1609406919899220

Parfenova, A., Marfurt, A., Pfeffer, J., & Denzler, A. (2025). Text annotation via inductive coding: Comparing human experts to LLMs in qualitative data analysis. In Findings of the Association for Computational Linguistics: NAACL 2025 (pp. 6471–6484). https://aclanthology.org/2025.findings-naacl.361/

a person lying on their back in a grassy area with trees in the background
Photo by Rom T on Unsplash

Discussion about this post

User's avatar

Keep reading

Anthralytic’s Substack

Anthralytic Substack

  • About
  • Archive
  • Recommendations
  • Sitemap
© 2026 Anthralytic · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.