Cohen's Kappa

Verified

Reliability (legacy hub)

Independently verified. Every statistic this test reports has been re-derived against an independent reference — never the library the pipeline itself calls — the rendered output was read back in a browser, and the result is locked with a committed regression suite.

Bring your own data and open the guided workspace: variable setup, assumption diagnostics, results with effect sizes and confidence intervals, figures, and APA-ready reporting.

Use your own dataset

Loading your datasets…

No data of your own? Try this test on a free teaching dataset →

Agreement between two raters on categorical judgments, corrected for the agreement expected by chance.

Cohen's kappa runs from below 0 (worse than chance) to 1 (perfect). Rough bands: .41–.60 moderate, .61–.80 substantial, > .80 almost perfect.

Worked example

Do two coders agree on classifying 100 responses?

Two coders independently assigned each of 100 responses to one of three categories; kappa corrects their raw agreement for chance.

Result

Agreement was substantial, κ = .73 (raw agreement 82%).

How you'd report it (APA)

Inter-rater agreement was substantial, Cohen's κ = .73.

Try it yourself: Load this ready-made sample and follow the run above.

When to use it

  • Two fixed raters, nominal categories
    The same two coders classify every case into unordered categories. e.g. two coders assigning 100 responses to one of three themes.
  • You need chance-corrected agreement
    Raw percent agreement flatters raters when one category dominates; kappa subtracts the agreement expected by chance.

When NOT to — use instead

  • Three or more raters
    Kappa is defined for two; generalise to a panel. → Fleiss' kappa
  • Ordered categories
    Nominal kappa treats a one-grade miss the same as a far miss; weight the disagreements. → Weighted kappa
  • Continuous measurements
    For numeric ratings use an agreement coefficient for continuous data. → Intraclass correlation

Assumptions (and what to do if they fail)

The kappa paradox with skewed marginalshigh

Check: When one category is very common, kappa can be low even at 95% agreement. Always report percent agreement and the marginal distribution alongside κ.

If violated: A genuinely reliable coding scheme is dismissed as unreliable purely because the trait is rare.

Same two raters across all casesmedium

Check: Cohen's κ assumes the two raters are fixed; a rotating pool of coders needs Fleiss.

If violated: Mixing raters violates the model and the chance-agreement term is mis-estimated.

Categories are mutually exclusive and exhaustivemedium

Check: Each case falls in exactly one category for each rater.

If violated: Overlapping or missing categories make the agreement table ill-defined.

Ready to run a Cohen's Kappa on your own data?

Guided setup, automatic assumption checks, effect sizes, figures and an APA write-up.

Run this test →