You need to sign in or sign up before continuing.
Reflection — An Honest Take 8 min

Honest Take — Before You Begin

Honest Take — Module 8: Experimentation — Falsifiability at Low Traffic #

This is the module where I have to repeatedly tell you the truth about a thing much of the PM industry has decided to be dishonest about, and you'll find it deflating before it becomes liberating. The truth: at your scale, classical A/B testing is almost always the wrong tool. Most indie and early B2B products do not have the traffic to detect realistic effect sizes with statistical significance, and running A/B tests anyway is statistical theater that lends intuition decisions a false confidence. The literature ignores this because the literature is mostly written by and for people at Stripe-scale, where the traffic supports the method. Their methodology is unimpeachable. The failure is importing it into a product with 1/1000th the traffic and presenting it as the same discipline. It isn't.

The deflation has a precise shape. Engineers come into PM with the correct intuition that hypotheses should be tested and data should drive decisions, and Module 8 does not reverse that intuition — it refines the operationalization. The right tools at your traffic level are pre-registration of predictions, sequential decision rules with rollback criteria, qualitative N=5-10 observation, and honest single-shot bets. None of these gets the marketing budget of an A/B-testing SaaS, and all of them are statistically valid where classical A/B testing at your scale usually isn't. It is classical A/B testing that is the special case — it requires large stable traffic, narrow effect-size detection, patience for sample size, and uncontaminated cohorts — not these other modes. The industry has the hierarchy upside down.

Pre-registration is the most important habit here and the most counter-intuitive, so let me handle the objection you're already forming: "I don't know what to expect; that's why I'm running the experiment." Intuitive, and wrong. The point of pre-registration is not to predict accurately; it is to commit to a falsifiable claim before the data arrives. Without the commitment, every result is interpretable; with it, you have either a confirmed prediction (you knew something) or a falsified one (you learned something). Both outcomes are valuable. What pre-registration kills is the unfalsifiable narrative — "the data was instructive in many ways" — which is what most product decisions accumulate in its absence.

And here is the specific prediction: your first pre-registered experiment will land between what you predicted and the null, the middle outcome will feel ambiguous, and you'll be tempted to write an interpretive paragraph about "directionally encouraging" data. Don't. The honest reading is "the prediction was wrong, and the change might still be worth keeping on a separate argument." Hold those two judgments apart; conflating them corrupts both your calibration and your decision.

Operationally: read Evan Miller's essays — "How Not To Run an A/B Test" especially — and run his sample-size calculator on your own product's traffic just to know the gap between the experiment you'd want and the traffic you have. The gap will likely be 10x or worse, and the number is the cheapest sobriety available. Absorb Gelman's garden-of-forking-paths idea — experimental flexibility itself manufactures false positives — and then write the decision-tree checkpoint: which modality (pre-registered bet, sequential rollout, qualitative N=5, classical A/B) you'd pick for which class of decision. Without the written tree, your modality choice drifts toward whichever tool you're comfortable with; with it, the choice is deliberate. The tree is the meta-tool, and it will be revised two or three times as your context shifts, which is itself useful history.

There's a moment coming in this module that I want you braced for: you will realize that most of the past decisions you congratulated yourself on as "data-driven" were intuition decisions wearing statistical theater. The realization is uncomfortable. Don't fight it, and don't over-correct into despising intuition — some of those decisions were good because your engineering and product instincts are real; what was false was the costume. The honest reckoning is what makes the next decade of decisions better. The dishonest version — continuing to call intuition "data-driven" — compounds into a practice that pretends to be evidence-based while running on vibes, and the pretense is a bigger problem than the vibes.

Conclusion #

Module 8 deflates the experimentation cargo cult and replaces it with the underlying discipline: pre-register, write decision rules, run the power math, use qualitative methods without apology, and reserve classical A/B testing for when traffic actually supports it — which, this year, it mostly won't. That is the right answer, not a consolation prize. Falsifiability is the discipline; the tooling is contingent.

Predictions #

  • Evan Miller's calculator, run on your real traffic, will show at least a 10x gap between the sample you'd need and the sample you have. The number will be sobering and clarifying.
  • Your first pre-registered prediction will be wrong in direction or magnitude. Good — the honest record of being wrong is what calibrates the next one.
  • Within a week of the module you'll be tempted to declare victory on an ambiguous result. The discipline is ternary: confirmed, falsified, or insufficient data. "Directionally encouraging" is the failure mode.
  • The don't-peek rule will be harder than it sounds; you will peek at least once. Note when you peeked and what you saw — data about you, if not about the experiment.
  • The forecasting habit will leak into your engineering work, most usefully into project-time estimates, where it will expose a correctable calibration pattern within a month or two.
  • At some point you'll have the "I've been calling intuition data-driven for years" reckoning. It will be uncomfortable and then freeing, in that order.