Hiberius

Incrementality Testing — Testing skill for Claude Code

Testing community

Agent Skill for causal measurement in marketing: sample ratio mismatch, an always-valid sequential test that survives daily peeking, CUPED variance reduction, sample sizing, and geo/holdout designs fo.

How to install Incrementality Testing

This entry records only its repository, not the path inside it, so there is no exact command to give. Open Hiberius/incrementality-testing and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Incrementality Testing does

Alternatives in Testing

Webapp Testing — Test local web applications using Playwright for UI verification and debugging 94.1k ★
Fix Issue — by metabase - Addresses GitHub issues by taking issue number as parameter, analyzing context, implementing sol 46.5k ★
Playwright — claude-plugins-official Browser automation, E2E testing, screenshots 29.4k ★

README

Incrementality Testing

**An Agent Skill for the causal question: did the ad cause the conversion, or take credit for it. Sample ratio mismatch, an always-valid sequential test that survives daily peeking, CUPED variance reduction, and the geo and holdout designs for channels where you cannot randomise users.**

[![License: MIT](https://img.shields.io/badge/License-MIT-2ea44f.svg)](LICENSE) ![Python 3.8+](https://img.shields.io/badge/python-3.8%2B-3776AB?logo=python&logoColor=white) ![Zero dependencies](https://img.shields.io/badge/dependencies-0-6E56CF)

npx skills add Hiberius/incrementality-testing

The same data, two legal answers

python3 scripts/lift.py lift --control 50120/1204 --treatment 49880/1330

absolute lift  +0.2642 pp
relative lift  +11.00%   95% CI [+2.89%, +19.11%]
p = 0.00787  significant

python3 scripts/lift.py msprt --control 50120/1204 --treatment 49880/1330

always-valid p        0.17168
decision              keep collecting

Both are correct. The first is what you may claim if you committed to that sample size in advance and looked once. The second is what you may claim if you have been watching it daily. **Choosing the first after seeing that it is smaller is the bias itself.**

Checking a fixed-horizon p-value every day and stopping the first time it dips under 0.05 pushes the real false positive rate to around 30%. That is the main reason A/B results do not replicate, and it is entirely self-inflicted.

The test suite proves the fix: twenty peeks under a true null, 200 simulated experiments, false positive rate stays at 1%.

Check the split before you read anything

python3 scripts/lift.py srm --arms "control=52000,treatment=48000"

SAMPLE RATIO MISMATCH. Stop. Do not read the lift.
The assignment, the logging or a filter is broken. A lift computed on a
broken split is not wrong by a little, it is meaningless.

Threshold 0.001, not 0.05. It is a smoke alarm,