Inter-Rater Reliability

Coming soon

Psychometrics (legacy hub)

This test is implemented and is currently going through StatMinds’ production verification: every statistic is independently checked against a trusted reference (scipy / R), locked with regression tests, and the screen is exercised across assumption-met/violated and significant/non-significant scenarios before it opens up.

See what’s live now

Quantifies agreement between two or more raters scoring the same subjects.

Reports the appropriate statistic for the scale: Cohen's κ for two raters with nominal categories, weighted Cohen's κ for two raters with ordinal categories (linear or quadratic weights), Fleiss' κ for ≥ 3 raters with nominal categories, and intraclass correlation (ICC, specifically ICC(2,k) two-way random-effects average-rater) for continuous ratings. Engine auto-routes by scale + n_raters and reports point estimate + bootstrap 95% CI + agreement-interpretation tier (Landis-Koch benchmarks).

Worked example

Do two raters agree when scoring the same responses?

Two raters independently scored 50 responses; inter-rater agreement is quantified with an intraclass correlation (ICC), or kappa for categories.

Result

Agreement was excellent, ICC = .88, 95% CI [.80, .93] — the scoring is reliable across raters.

How you'd report it (APA)

Inter-rater reliability was excellent, ICC = .88, 95% CI [.80, .93].

When to use it

  • Two raters, categorical outcome (nominal or ordinal)
    Two radiologists rate 80 X-ray scans as normal / mild / severe.
  • ≥ 3 raters, continuous ratings (ICC two-way random)
    5 graders score 60 student essays on a 1-100 scale.

When NOT to — use instead

  • Internal consistency (within a single rater across items)
    Cronbach's α is between-item, not between-rater. → Cronbach's Alpha
  • Test-retest (same rater scoring same subjects twice)
    Test-retest is single-rater stability over time — use ICC(3,1) two-way mixed (rater fixed). → Inter-Rater Reliability
  • Single rater scoring on a single occasion
    No agreement to estimate — needs ≥ 2 raters or 2 occasions. → CTT Item Analysis
  • Diagnostic accuracy against a gold standard
    Agreement between rater and TRUTH is sensitivity / specificity / AUC, not κ. → Cut-Offs (ROC)

Assumptions (and what to do if they fail)

Raters are exchangeable (same construct + same training)medium

Check: See the assumption diagnostics in the workspace.

If violated: The workspace flags this and suggests a robust or nonparametric alternative.

Each row is an independent subject (no clustering)medium

Check: See the assumption diagnostics in the workspace.

If violated: The workspace flags this and suggests a robust or nonparametric alternative.

For weighted κ: categories are ordered with meaningful distancesmedium

Check: See the assumption diagnostics in the workspace.

If violated: The workspace flags this and suggests a robust or nonparametric alternative.

For ICC(2,k): raters are a random sample from a larger population of ratersmedium

Check: See the assumption diagnostics in the workspace.

If violated: The workspace flags this and suggests a robust or nonparametric alternative.

Ready to run a Inter-Rater Reliability on your own data?

Guided setup, automatic assumption checks, effect sizes, figures and an APA write-up.

Run this test →