Whose Meaning Survives the Analysis?
Every step from a recording to a finding drops something, and what gets dropped isn’t random.
Part 4 of 6
This is the fourth piece in a six-part series on AI and qualitative interpretation. Part 1 asked when AI strengthens qualitative interpretation and when it reproduces, amplifies, or obscures human biases. Part 2 asked what a model matching human coders actually shows. Part 3 asked when working with AI exposes the assumptions a team shares and when it reinforces them. This piece takes up the third subquestion: whose meanings are preserved, changed, or lost as material becomes codes, themes, and findings?
Here is another hypothetical. In a needs assessment at a refugee camp, a woman mentions, in passing and in her second language, that she doesn’t send her daughter to the water point after dark anymore. She isn’t complaining. She’s explaining why the household uses less water than it did. The interviewer moves on.
That sentence contains a protection concern. Nobody said “safety.” Follow it through the analysis and ask at each step whether it’s still there. It has to survive the recording, the transcription, the decision about which documents get reviewed, the coding, the grouping of codes into themes, and the synthesis into something a program manager can act on. Each step is a transformation, and each transformation is a chance for the sentence to disappear. This piece follows the material through those steps, with the evidence on what gets lost at each one, and with both human choices and model behavior in view.
Loss starts before anyone interprets anything
The first transformation is from speech to text. In From Algorithms to Evidence, Lea Corsetti, Edwin Asare, and Diana Cekova describe a 2024 evaluation of CGIAR’s science groups that ran more than 350 interviews through AI transcription. It saved weeks. They also report that accuracy fell for non-native accents and noisy environments, and that every set of notes needed full human review.
Transcription tools have improved since 2024, and the chapter is careful to present this as that year’s experience. The structural point holds regardless of the tool: transcription errors fall unevenly. They fall on the speakers whose accent or language or recording conditions the system handles worst, and those are often the people a needs assessment most needs to hear. Transcription accuracy isn’t interpretive accuracy, but you can’t interpret a sentence that didn’t make it onto the page.
What isn’t documented gets mistaken for what didn’t happen
The second transformation is selection: which material gets analyzed at all. Azra Nurkic and Lauren Ropp describe a thematic review of a donor’s internal reports, more than 4,000 documents from 296 activities, narrowed to 197 activities for detailed review. They built a careful system: a closed document library, a codebook, standard prompts asking for short summaries with exact citations, and a review step where every AI-generated row was marked Accept, Edit, or Discuss.
The problem they name most clearly isn’t the model. It’s documentation bias. Some activities had many detailed reports; others had few or thin ones. In their words, “the tool can only surface what exists,” so the well-documented activities produced more tags and fuller summaries. An activity with thin reporting looks, in the analysis, like an activity where less happened. They responded with practical safeguards: a flag for documentation depth, counts normalized per report rather than per activity, and targeted human reading of thin cases before synthesis. Those are good responses, and they’re responses to a bias in the source material that a model would faithfully reproduce. They reduce the distortion. Nothing in the chapter claims they removed it, and nothing could.
Coding errors aren’t random about who is speaking
The third transformation is coding, and here the evidence is sharpest. Ashwin, Chhabra, and Rao compared language-model coding of interviews with Rohingya refugees and their Bangladeshi host community against careful human annotations. The models’ errors were, in the authors’ words, “not random with respect to the characteristics of the interview subjects.” For one model, the errors tracked household characteristics such as refugee status, education, and income on 10 of 19 codes. An error pattern like that doesn’t just lower accuracy. It changes what the study would conclude about those groups, because the people whose words were most often miscoded are also the people the conclusions are about. The comparison rests on human codes, which can’t by themselves show how biased the humans were, but the shape of the model’s errors is the finding, and it stands.
A 2026 preprint from Marston and colleagues shows what that pattern looks like theme by theme. They benchmarked 46 models on 150 synthetic transcripts written to resemble refugees’ answers to a needs question, against two experienced human coders whose disagreements were adjudicated by a lead researcher. Thirty-two models cleared the usual reliability threshold. Then the authors looked at which codes the errors clustered in. Physical safety was a source of disproportionate errors for 13 of the 14 models they examined closely. Several models tended to recode accounts of discrimination as the service that was denied, so a person turned away from a clinic because of who they were became a health need. And no model was consistently good at recognizing a need that fell outside the nine predefined categories. The authors’ explanation is that needs expressed through implication, narrative subtext, or culturally specific framing are systematically harder for the models. That’s the woman at the water point. The transcripts were synthetic, so how this holds up in the field is open. The pattern is what matters: strong aggregate performance sat beside the specific omissions that would matter most to a protection officer.
Repetition gets mistaken for significance
The fourth transformation is grouping and counting. The first piece in this series opened on Geordie Fung’s case, in which a model analyzing a think tank’s publications treated regional partnerships as central because they came up constantly and underweighted budgetary trade-offs that mattered more. The point to add here is about the source material. Institutional documents are written by the institution. They repeat what it values, in the language it uses, and they’re silent about what it doesn’t track. A model that counts themes in those documents will faithfully report the institution’s habits as findings. So will a human analyst who isn’t watching for it. The frequency problem isn’t a model quirk. It’s what happens when anyone, human or model, lets the volume of mentions stand in for their weight.
The last step is turning accounts into something an organization can use
The final transformation is synthesis, and it happens under pressure. Trevor Smith and Kathryn Kraft describe VisionFund International’s effort to analyze open-ended survey responses from about 15,000 microfinance clients, in multiple languages, using ChatGPT to propose themes and suggestions. Their stated aim is to elevate client voices. They name the tension in their own method: a phenomenological approach wants the client’s account to speak for itself, while the organization wants insights it can act on. Those pull in different directions. The model’s summaries served the second purpose well and the first less well, and the authors say so: an untrained tool “loses some of the nuance necessary to accurately reflect client feedback,” and global insights stripped of context “may actually mitigate client voices rather than elevate them.” They checked about 15 percent of the analysis by hand and found it broadly consistent, which tells you the model applied the themes the way the team would have. It can’t tell you whether the team and the model were flattening the same things.
The software is part of the analysis too
One more influence runs through every stage, and it’s easy to miss because it looks like plumbing. Kathryn Wheeler analyzed the same set of young people’s open-ended survey responses about the future in three environments, MAXQDA, NVivo, and ChatGPT, and kept a reflexive record of how each one shaped what she could see. MAXQDA kept each participant’s responses together; NVivo returned her to single cells; ChatGPT produced a tidy typology that, in her words, “imposes coherence where there may be none” and left many one-word answers uncoded. She calls the practice of noticing this technological reflexivity: examining how the tools’ architecture, segmentation, and summary formats shape interpretation, alongside the researcher’s own position. It’s one researcher’s situated account, not a controlled comparison, and that’s its value. It shows that what was made visible as analytically meaningful had already been filtered before she read it.
What research would need to follow
The studies above each catch one transformation. Research on this question would need to follow the whole chain and ask, at each step, what dropped out and for whom. That means tracking omissions by the speaker’s language and characteristics, by the quality of the source documents, and by the type of account: direct complaint, indirect mention, contradiction of the emerging theme. It means assessing the consequences for the conclusions and the decisions that follow, not just the code-level accuracy. And it means asking a question none of these studies was designed to answer: are the humans and the models missing the same accounts, or different ones? If different, a model might catch what a tired analyst skipped. If the same, the review step confirms the omission instead of finding it.
Back at the water point, the question is whether anyone recorded that sentence, whether it survived transcription in a second language, whether a code existed for it, whether it was grouped with the safety concerns or with the water ones, and whether it reached the program manager as a protection issue or as a drop in water use. Every one of those is a human decision or a model behavior that could be studied.
What survives the analysis is whatever the chain was built to carry. The next piece asks how humans and AI might check each other’s work along the way, and what that checking would need to look like to catch what both are inclined to miss.
Anthralytic builds evaluation tools for organizations that can’t afford an evaluator on staff. Evaluability checks, theory of change builders, and planning assistants, designed so the people doing the work can do their own evaluation, with an evaluator in the loop when it counts.
References
Ashwin, J., Chhabra, A., & Rao, V. (2025). Using large language models for qualitative analysis can introduce serious bias. Sociological Methods & Research. https://doi.org/10.1177/00491241251338246 (open access)
Corsetti, L., Asare, E., & Cekova, D. Experimenting with AI tools in CGIAR Science Group evaluations: A candid assessment of early adoption. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139
Fung, G. Designing with generative AI: Using GPT-4 to build a monitoring, evaluation, and learning framework. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139
Marston, J., Kreutzer, T., Garnier, S., Boone, E., Pham, P. N., & Vinck, P. (2026). Can large language models reliably code qualitative humanitarian data? A benchmark study against human expert adjudication [Preprint]. arXiv. https://arxiv.org/abs/2606.26541
Nurkic, A. K., & Ropp, L. Integrating generative AI into thematic analysis: A structured approach for large-scale document review in evaluation. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139
Smith, T., & Kraft, K. Elevating local voices: Qualitative analysis reimagined. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139
Wheeler, K. (2026). Technological reflexivity in practice: How MAXQDA, NVivo, and ChatGPT shape qualitative survey analysis. Qualitative Research in Psychology. Published online 31 January 2026. https://doi.org/10.1080/14780887.2025.2602820 (open access)

Discussion about this post