Experimentation — Falsifiability at Low Traffic

Build experimental rigor sized to your actual traffic: understand the statistics of A/B testing well enough to know that most experiments you'll want to run are statistically underpowered at your scale — and learn what works instead: pre-registration, sequential decision rules, qualitative small-N work, and calibrated forecasting. By the end you can design a small experiment, set the decision rule before running it, and refuse to dignify intuition decisions with the language of data. A small-scale experiment is a canary deployment: the change is the deploy, the rollback criterion is the canary, the window is the stability check — most experimental discipline at your scale is canary-style operational rigor, not hypothesis testing. Pre-registration is test-first, again: commit to expected behavior before observing actual. Power analysis is load-test sizing — most founders skip it for the same reason teams skip load-test sizing: it requires admitting upfront whether the effort is worth doing. Not peeking is not deploying mid-test. And decision rules are runbooks: responses written calmly in advance beat decisions improvised under pressure, in incidents and in experiments alike.