Intercoder agreement is useful in some qualitative designs, particularly structured content analysis with a defined codebook and independent coding decisions. It is not a universal measure of quality. In reflexive thematic analysis and other interpretive approaches, discussion, reflexivity, and transparent development of meaning may be more methodologically coherent than seeking a single agreement statistic.
Choose the workflow to fit the method
Independent coder copies protect separation before comparison. Shared live work supports dialogue, co-construction, and documented consensus. These are different purposes and should not be reported as the same practice.
Interpret statistics cautiously
Agreement coefficients depend on the unit, code prevalence, missing decisions, and calculation rule. A number alone does not establish that a coding scheme is conceptually valid or that researchers understood the material in the same way.
Report the process
Explain coder roles, independence, training or calibration, the material compared, disagreements, and how changes affected the codebook. Do not imply reliability work occurred if the project used collaborative interpretation instead.
Questions researchers often ask
Is disagreement a problem?
It can reveal unclear definitions, competing interpretations, or important ambiguity. Its meaning depends on the study’s methodological stance.