Paired Permutation Test (distribution-free)

Coming soon

T-Tests

This test is implemented and is currently going through StatMinds’ production verification: every statistic is independently checked against a trusted reference (scipy / R), locked with regression tests, and the screen is exercised across assumption-met/violated and significant/non-significant scenarios before it opens up.

See what’s live now

Distribution-free randomization analogue of the paired t-test.

The observed mean difference (or |t|) is referenced against a null distribution built by randomly flipping the signs of the paired differences, producing an exact / Monte-Carlo p-value that does not rely on the normality of the difference scores.

Worked example

Is a paired difference real without distributional assumptions?

A paired permutation test rebuilds the null by randomly flipping the sign of each pair's difference thousands of times.

Result

The mean difference was significant, permutation p = .004 (10,000 resamples).

How you'd report it (APA)

A paired permutation test showed a significant mean difference, p = .004 (10,000 resamples).

When to use it

Pre/post or matched-pairs with continuous DV
25 patients have pain scores measured before and after 6 weeks of physical therapy.
Small-sample paired design with outliers
Pilot biomarker study: 12 patients measured pre/post intervention.

When NOT to — use instead

Independent groups
Permutation t (independent) shuffles GROUP LABELS, not signs. → Permutation t-test (distribution-free)
Three or more related measurements
Sign-flip is for k = 2 paired conditions. → One-Way RM-ANOVA (3+ measurements)
Paired binary outcome
Sign-flip applies to numeric differences. → Exact McNemar's Test (small-sample binary paired)
Symmetric difference distribution holds (paired-t valid)
Paired-t is more powerful when assumptions hold — use it for parsimony. → Paired Samples t-test

Hypotheses

H₀: the distribution of paired differences is symmetric around zero — signs of d_i are equally likely + or −.

Hₐ: the distribution of paired differences is not symmetric around zero.

Parameter tested: paired mean difference

Assumptions (and what to do if they fail)

Adequate number of paired observations (≥ 10 recommended) for the Monte-Carlo permutation null.medium

Check: Report n_pairs; if below 10, flag.

If violated: Very small n makes the Monte-Carlo permutation distribution coarse; the minimum two-sided p is 2/2^n.

Paired differences should not collapse predominantly to zero (zero differences carry no information under the sign-flip permutation).low

Check: Count zero differences.

If violated: Zero differences contribute no information under a sign-flipping permutation scheme.

Ready to run a Paired Permutation Test (distribution-free) on your own data?

Guided setup, automatic assumption checks, effect sizes, figures and an APA write-up.

Run this test →

🔎 Worked example

✓ When to use it

✗ When NOT to — use instead

Hypotheses

Assumptions (and what to do if they fail)

Worked example

When to use it

When NOT to — use instead