Permutation t-test (distribution-free)

Coming soon

T-Tests

This test is implemented and is currently going through StatMinds’ production verification: every statistic is independently checked against a trusted reference (scipy / R), locked with regression tests, and the screen is exercised across assumption-met/violated and significant/non-significant scenarios before it opens up.

See what’s live now

Tests the null of NO group-label-vs-outcome association by computing the t-statistic on every permutation of the group labels (or a Monte-Carlo subset, default 10,000 reshuffles), then comparing the o

bserved t to that empirical null distribution. The resulting p-value is exact under exchangeability and makes NO normality assumption — the right tool when the parametric t's distributional assumptions are clearly violated AND the sample is too small for the CLT to bail you out.

Worked example

Do two groups differ without assuming normality?

A permutation t-test rebuilds the null by shuffling group labels — valid for small or non-normal samples.

Result

The groups differed by 5.1 points, permutation p = .013 (10,000 shuffles).

How you'd report it (APA)

A permutation t-test showed a significant difference between groups, p = .013 (10,000 permutations).

When to use it

  • Small-sample two-group comparison (n per group < 30)
    A pilot study compares a new surgical technique vs.
  • Non-normal continuous DV at any sample size
    Comparing income (right-skewed, dollars) between two occupational groups (n = 80 vs 75).

When NOT to — use instead

Hypotheses

H₀: the two groups are drawn from the same distribution (under exchangeability, labels are interchangeable). With similar shape/spread, this specialises to H₀: μ₁ = μ₂.
Hₐ: the two distributions differ. With similar shape/spread, this specialises to μ₁ ≠ μ₂.

Parameter tested: population mean difference μ₁ − μ₂ (interpretable as a location shift when shapes are similar)

Assumptions (and what to do if they fail)

Under the null hypothesis, the two groups are exchangeable — each observation could equally have come from either grouphigh

Check: Exchangeability follows from random assignment / random sampling.

If violated: If exchangeability fails, the permutation p-value no longer reflects the true null distribution.

The two groups have similar distribution shapes and spreads (for a pure location-shift interpretation)medium

Check: Per-group boxplots with similar IQRs and symmetric shapes suggest a pure location shift.

If violated: When shapes/spreads differ, the permutation test still gives a valid p-value for the null 'same distribution', but the mean-difference interpretation is less clean.

No extreme outliers within either groupmedium

Check: Per-group boxplot inspection.

If violated: Outliers inflate the mean difference; the permutation test is less affected than the parametric t, but extreme points can still dominate.

Ready to run a Permutation t-test (distribution-free) on your own data?

Guided setup, automatic assumption checks, effect sizes, figures and an APA write-up.

Run this test →