Module 10 Reporting & testing
A/B vs lift studies
Split people into two groups, or hold a group back and measure what the ads added.
Both designs compare groups of people
An A/B test splits people into two groups and shows each group a different ad. A lift study keeps a group out of the ads on purpose, so you can see what would have happened without them. Both are experiments. The difference is whether the second group sees a different ad, or no ad. One design compares ads. The other compares ads with nothing.
An A/B test splits the audience
In a Meta A/B test each version is shown to a segment of your audience, ensuring that nobody sees both, and the test then decides which version performed best. Google Ads experiments split traffic between a control group or campaign and one or more treatment groups. TikTok Split Testing keeps each audience group to one ad group so the two do not compete. An A/B test answers “which version?”
A lift study holds people back
Meta’s Conversion Lift divides the audience into a test group who have the opportunity to see the ads and a control group who do not, and the difference in conversions between them reveals the lift. Google Ads reports Conversion Lift against an unexposed holdback group. TikTok’s Conversion Lift Study compares a test group and a control group the same way. A holdout measures what the ads added, not which ad won.
Some tests do both at once
Google reports A/B plus Conversion Lift, which measures per-arm holdback incrementality and gives head-to-head winner scores between the arms. Meta says you can add a holdout to an A/B test to compare the cost per incremental purchase of each campaign. Here the two ad groups each shrink so a slice can be held back. You can compare versions and measure lift in one study.
Same design, different names
Google Ads has experiments, Conversion Lift and Brand Lift. Meta has A/B tests, Conversion Lift and brand survey tests that measure Brand Lift. TikTok has Split Testing, Conversion Lift Studies and Brand Lift Studies. Pinterest lets you create an A/B test with up to four versions. Snapchat documents Split Testing and Brand Lift. Translate the test name before you compare the result.
| Sample | Value |
|---|---|
| People in the sample | 240 |
| Held back | 20% |
| Modes | A/B split · Holdout · A/B + holdout |
| Sample | Value |
|---|---|
| People in the sample | 240 |
| Held back | 20% |
| Modes | A/B split · Holdout · A/B + holdout |