Can AI Help Us See the Assumptions We Share?
Agreement that comes easily may be clarification or a shared blind spot, and from the inside they look the same.
Part 3 of 6
This is the third piece in a six-part series on AI and qualitative interpretation. Part 1 asked when AI strengthens qualitative interpretation and when it reproduces, amplifies, or obscures human biases. Part 2 asked what we’ve learned when a model matches human coders, and ended on the thought that a model that matches our codes has learned how we read. This piece takes up the second subquestion: when does working with AI expose the assumptions a team shares, and when does it reinforce them?
Here is a hypothetical. A team is evaluating a workforce program. Over several rounds of interviews they’ve coded “barriers to participation” as transport, childcare, and scheduling. They give the model their codebook and a stack of new transcripts. It finds the same three barriers everywhere, cleanly applied, and nothing else. The team agrees with every code. The report writes itself.
Nothing in that process could have surfaced a fourth barrier. The model was given three and found three. The team already believed in three and saw them confirmed. Everyone is in the water, and the water is invisible. The convergence feels like confirmation, and it may well be. The question this piece is about is how you would know.
Teams already have ways to catch this, and they work partly
Qualitative researchers have a name for the practice of examining your own role in what you find: reflexivity. Olmos-Vega and colleagues define it as a set of continuous, collaborative practices through which researchers examine how their position shapes the work, and they sort it into four kinds. Personal: your background and assumptions. Interpersonal: the relationships in the study, including power within the team. Methodological: whether your choices fit your framework. Contextual: how the setting shapes what you can see.
Done as a team, reflexivity has teeth. Barry and colleagues describe a health research team that wrote reflexive accounts of their personal agendas, hidden assumptions, and theoretical definitions, shared them, and discussed what came up. The exercise surfaced differences the team hadn’t known it had. Codebook revision, which came up in the last piece, does similar work: a disagreement over a code forces people to say what they think it means.
The common thread is that these practices expose assumptions by making people articulate them to someone who might not share them. That’s the function a model would have to perform to help here. The question is whether a model can perform it, or only appear to.
A model doesn’t disagree the way a colleague does
A colleague who disagrees with you has a reason and a stake. They read the transcript differently because of what they know and who they are, and they’ll hold their ground until you persuade them. A model that agrees with you may simply be reflecting your prompt.
The research on model behavior gives this a name: sycophancy, the tendency of AI assistants to give answers that match what the user seems to believe rather than what the evidence supports. Sharma and colleagues found it across all five of the leading assistants they tested in 2023 and traced part of the cause to how the models are trained: human raters tend to prefer responses that agree with them, and the models learn to oblige. The models have changed since; the training pressure hasn’t.
Evaluators have noticed the same thing from the practice side. In From Algorithms to Evidence, Lauren Damme describes building a custom GPT to support early-career evaluators in designing rigorous evaluations, then discovering there was no validated way to assess whether such a tool helps or harms the people who use it. With a group of experienced evaluators, she drafted a rubric. Its fifth criterion is confirmation bias and sycophancy, which she defines as the model mirroring the user’s framing in ways that limit their consideration of other perspectives. Her first criterion asks whether the tool helped or harmed the user’s ability to think critically. The rubric is a proposed test, not a finding that any tool passed or failed it, and that’s exactly what makes it useful here. It names, in practitioners’ language, what a study of AI and shared assumptions would need to measure.
But the model can also put something on the table that nobody in the room would have
The opposite case deserves equal weight, and the book has one. Hailemichael Taye Beyene describes a doctoral evaluation of progress toward ending poverty across eight sub-Saharan African countries, using fuzzy-set qualitative comparative analysis, a method for testing which combinations of conditions go together with an outcome across a set of cases. Before the analysis could run, he had to reduce a list of more than seventeen overlapping factors (social protection, governance quality, aid dependence, and so on) to a smaller set of clearly defined conditions. He was working alone, with no team to argue with.
He used ChatGPT as what he calls a surrogate partner. It contrasted definitions, proposed alternative groupings, suggested terminology, and ran what-if combinations. He describes the result as reducing the isolation of the task and prompting him to revisit his own assumptions. He also describes the costs in the same breath: every suggestion had to be checked against theory and primary evidence, and small wording changes accumulated into what he calls interpretive drift, where a condition’s meaning shifted without anyone deciding it should.
This is conceptual structuring, not interview coding, so the lesson transfers as a possibility rather than a result. But the possibility is real. In Beyene’s account, the model did what Barry’s colleagues did for each other: it put alternatives on the table that one person, working alone, might not have generated. Whether a model does this reliably, and under what conditions, is the research question. His account shows the question is worth asking.
The two studies that have looked at the exchange itself
Both of these studies came up briefly in the first piece. Here is what they actually found.
CoAIcoder ran 32 pairs of novice coders through four arrangements: coding independently with no AI, each coder with their own AI suggestions, a shared AI model that learned from both coders, and that shared model with both coders working at the same time. The AI was a classifier trained on the coders’ own history, not a chatbot. The shared-model conditions were faster in the first coding phase and reached higher early agreement. They also produced fewer distinct codes: about ten per pair in the shared asynchronous condition against fifteen with no AI. The authors’ explanation is independence. When a shared model carries one coder’s choices to the other, the two are no longer reading independently, and the range of what they notice narrows.
Reflexis was built for the opposite purpose. Designed for reflexive thematic analysis, it uses a language model to summarize each coder’s reasoning, generate discussion prompts that take the coders’ stated positions into account, and flag when a code seems to be drifting from its original meaning. It doesn’t suggest codes. Twelve researchers in six pairs used it for one session each on synthetic transcripts, with no comparison condition. They reported more detailed reflection and said disagreements turned into productive conversations. The authors are clear that this speaks to first impressions, not to changed practice.
Reduced code diversity isn’t proof of bias; the pairs may have dropped redundant codes. Reported reflection isn’t proof of rigor. What both studies show is narrower and more useful: the arrangement of the collaboration changes what comes out of it, and that means the arrangement can be studied.
A second model isn’t a second opinion
That finding points at what shapes the range of interpretations. The prompt carries the team’s framing. The codebook carries it. The discussion that happened before anyone opened the model carries it. A different model, given the same prompt and the same codebook, isn’t an independent perspective. It’s the same framing through a different mirror. O’Connor and Joffe make the human version of this point when they note that coders can converge through what they call interpretive convergence rather than through insight. Adding a model to a team that has already converged doesn’t restore independence. It can remove the last of it.
Independence is a property of the process, not a count of participants.
What research would need to do
The studies above compare finished outputs or report first impressions. The question this piece asks lives in the middle, and research would need to look there. Record each coder’s initial interpretation before anything is shared, human or model. Log every revision with its reason. Look at the alternatives that were raised and rejected, and find a way to ask about the ones never raised at all. Vary the teams, the contexts, and who holds authority over the codebook, because a junior coder’s willingness to disagree with a model may depend on whether the senior coder set it up.
Back in the workforce evaluation, the test is simple to state. Did anyone, human or model, ever propose a fourth barrier? If so, what happened to it? If not, the clean report tells you the team and the model agreed. It can’t tell you what they agreed to overlook.
Agreement is cheap to produce and expensive to check, and the expensive part is where the assumptions live. The next piece follows the material itself, from recording to findings, and asks whose meanings survive the trip.
Anthralytic builds evaluation tools for organizations that can’t afford an evaluator on staff. Evaluability checks, theory of change builders, and planning assistants, designed so the people doing the work can do their own evaluation, with an evaluator in the loop when it counts.
References
Barry, C. A., Britten, N., Barber, N., Bradley, C., & Stevenson, F. (1999). Using reflexivity to optimize teamwork in qualitative research. Qualitative Health Research, 9(1), 26–44. https://doi.org/10.1177/104973299129121677
Beyene, H. T. Bridging complexity with intelligence: The role of generative AI in a mixed-methods evaluation of SDG1 in Sub-Saharan Africa. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139
Damme, L. Augmenting early career evaluators: Responsible development of a generative AI tool for designing rigorous and equitable evaluations. In K. Bruce, V. J. Gandhi, & S. Bohni Nielsen (Eds.), From algorithms to evidence: Using GenAI in evaluation practice. Routledge. https://doi.org/10.4324/9781003799139
Gao, J., Choo, K. T. W., Cao, J., Lee, R. K.-W., & Perrault, S. (2024). CoAIcoder: Examining the effectiveness of AI-assisted human-to-human collaboration in qualitative analysis. ACM Transactions on Computer-Human Interaction, 31(1), Article 6. https://doi.org/10.1145/3617362 (open preprint: https://arxiv.org/abs/2304.05560)
O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19. https://doi.org/10.1177/1609406919899220
Olmos-Vega, F. M., Stalmeijer, R. E., Varpio, L., & Kahlke, R. (2023). A practical guide to reflexivity in qualitative research: AMEE Guide No. 149. Medical Teacher, 45(3), 241–251. https://doi.org/10.1080/0142159X.2022.2057287 (open access)
Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2023). Towards understanding sycophancy in language models [Preprint]. arXiv. https://arxiv.org/abs/2310.13548
Ye, R., Huang, O., Lee, P. Y. K., Liut, M., Nobre, C., & Kong, H.-K. (2026). Reflexis: Supporting reflexivity and rigor in collaborative qualitative analysis through design for deliberation. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI 2026). https://doi.org/10.1145/3772318.3791275 (open manuscript: https://arxiv.org/abs/2601.15445)

Discussion about this post