Paid Media Decoded

Module 10 Reporting & testing

Reading results honestly

Two conversion rates are rarely enough. Change the group size and watch the interval decide.

10 minBeginner5 steps

With 200 people, the result is mostly noise

Group A converts at 4%, group B at 5%. On 200 people per group that difference could easily be chance. The interval here runs from below zero to well above it, so the honest answer is “too close to call”, whatever the two rates look like side by side. A difference without an interval invites a wrong decision.

More people narrows the interval

Same two rates, now with 5,000 people per group. The difference is still one percentage point, but the interval has shrunk and no longer includes zero, so the gap is likely real. Nothing about the creative changed. Only the amount of evidence did. The size of the sample decides how much a gap can tell you.

A bigger gap is easier to believe

Back to 1,000 people per group, but now group A converts at 3% and group B at 6%. A three-point gap is easier to separate from noise than a one-point gap, so the interval clears zero again. Big effects show up in smaller tests. Small ones need more people. The smaller the effect you chase, the more people it needs.

What the interval is telling you

The line is the difference between the two rates, in percentage points. The bar is a 95% interval: if you repeated this test many times, about 95 of every 100 intervals built this way would contain the true difference. If the bar crosses zero, “no difference” is still in play. The arithmetic here is illustrative statistics, not a platform’s method. If the interval crosses zero, you have not shown a difference.

What each platform says about confidence

Meta says its A/B test statistically calculates a winner with a certain confidence percentage, the chance of similar results if the test were repeated, and its Conversion Lift guidance treats 90%+ as high confidence and below 50% as a case to retest. TikTok’s Split Testing is designed to run at a 90% confidence rate. Google Ads reports an experiment’s effect as a point estimate with a margin of error, which describes a confidence interval. Pinterest asks you to watch the estimated test power before you publish. Every platform has a confidence rule. Ask for it in writing.

Platform
Interactive diagram
Slider simulator · Default state
ResultValue
Difference1 percentage points
95% interval-3.0619788281082903 to 5.06197882810829
VerdictToo close to call

Analytics settings

Allow Google Analytics? It helps us see which lessons people use. You can change this in Analytics settings.

Search the course

Type to find lessons, terms and sources.