Sign inBook a demo
Guide to measurement

Measuring marketing when the numbers don't add up

Three methods can tell you what your marketing is doing: attribution, incrementality testing and marketing mix modelling. Each answers a different question and each has a blind spot. This guide covers the problems they solve, where to start, and how to use them together.

Ad platforms claimed255 sales
Meta120Google95TikTok40
The store recorded180 sales
At least 75 of those claims are the same sale counted twice. And none of the three dashboards can say how many of the 180 would have happened with no ads at all.Illustrative figures for one month.

Choose how much detail you want. You can switch at any point.

Everything you need to choose a method and use it well.

Part 1

The problems

Most frustration with marketing measurement comes down to seven problems. Tick the ones you recognize, and the rest of the guide will point out what addresses them.

Part 2

The three methods

Each method answers one question well. Trouble starts when a method is asked a question it wasn't built for.

Attribution

Which ads did this customer touch before buying?

Follows the clicks and views it can track before a purchase, then shares the credit for the sale between them.

Speed
Daily
Detail
Down to the ad
Shows cause
No
Sees offline
No

Good at

  • Day-to-day decisions on campaigns, creative and keywords.
  • Fast feedback when something breaks or takes off.

Falls short

  • Shows what came before a sale, which is different from what caused it.
  • Favours whatever comes last, such as branded search and retargeting.
  • Blind to what it can't track: offline sales, ads nobody clicked, people who decline cookies.

What you need

Consistent campaign tagging, working conversion tracking, and one place where every platform's data is counted by the same rules.

Technical note. Rule-based models (last click, linear, position-based, time decay) assign credit by formula. Data-driven models (Shapley values, Markov-chain removal effects) infer it from observed paths. Both are observational: neither estimates what would have happened without the ad.

Incrementality tests

Would this sale have happened without the ad?

Shows ads to one group and withholds them from a similar group. The difference in sales between the two is what the ads caused.

Speed
Weeks per test
Detail
One channel per test
Shows cause
Yes
Sees offline
Yes, with region tests

Good at

  • Cause and effect. It is the closest thing marketing has to a ground truth.
  • Results a finance team can follow without a statistics lesson.
  • Region-based tests need no individual tracking.

Falls short

  • Answers one question at a time.
  • A result is a snapshot. It ages as creative, season and spend change.
  • Costs something: weeks of waiting, and some sales in the group that went without ads.
  • Small channels may not produce enough sales to measure.

What you need

The ability to withhold ads from some regions or audiences, enough sales volume to see a difference, and a few weeks without other big changes.

Technical note. Two main designs: user-level holdouts, usually run inside a platform as a conversion lift study, and geo experiments, which randomize or match regions and compare total sales. Outputs are incremental conversions and incremental ROAS, each with a confidence interval.

Mix modelling

How much did each channel contribute, and what if we moved the budget?

Sets a few years of weekly spend and sales side by side and works out how far sales moved when each channel's spend moved, after allowing for seasons, prices and promotions.

Speed
Refreshed weekly to quarterly
Detail
Channel level
Shows cause
Partly
Sees offline
Yes

Good at

  • The whole picture in one view, including offline and brand activity.
  • Budget questions: where returns are flattening and where there is room.
  • Needs no cookies or user tracking.

Falls short

  • Needs history, and spend that has varied. If every channel always moved together, it can't tell them apart.
  • Works at channel level. It can't choose between ads.
  • It is a model. Different reasonable assumptions give different answers, so it needs checking against tests.

What you need

Ideally two or more years of weekly spend and sales by channel, plus dates for promotions, price changes and anything else that moved sales.

Technical note. A regression, often Bayesian, on aggregate time series. Media variables are transformed for carryover (adstock) and diminishing returns (saturation curves); controls cover seasonality, price, promotions and macro factors. Outputs are channel contributions, ROI with uncertainty intervals, response curves and budget scenarios.

Which method answers which problem

Select any cell to see the reasoning.

Proving what marketing contributes
Platforms claiming more than you sold
Credit for sales you'd have made anyway
Where the next dollar should go
Tracking getting worse
Sales and media that can't be clicked
What to change this week
Part 3

Where to start

Five questions about your situation, then a suggested order for adopting the methods.

  1. Question 1 of 5

    What do you most need to know?

  2. Question 2 of 5

    How many channels get a meaningful share of your budget?

  3. Question 3 of 5

    Do meaningful sales or media happen where clicks can't be tracked?

  4. Question 4 of 5

    How far back do your weekly spend and sales figures go?

  5. Question 5 of 5

    Could you pause or hold back ads in some regions or for some audiences for a few weeks?

  6. Your suggested starting point

A starting heuristic built on common rules of thumb. Your data and sales cycle may shift the answer.

Using each one well

Whichever method you start with, these habits separate numbers you can act on from numbers that only look precise.

Attribution

  • Use one model for every platform.Each platform's dashboard is marking its own homework. Bring the data into one place and apply the same rules to all of it.
  • Compare within a channel, not between channels.Attribution is dependable for ranking campaigns inside one platform. It is much weaker for choosing between platforms.
  • Be suspicious of the top performers.Branded search and retargeting usually lead the table because they arrive last. Test them before giving them more.
  • Choose an attribution window and keep it.Changing the window changes the result, so a figure only means something beside others measured the same way.
Technical note. View-through windows are usually the largest single source of inflation. Data-driven models spread credit more sensibly than position rules, but they still describe paths and don't estimate a counterfactual.

Incrementality tests

  • Test the biggest or most doubted spend first.A test costs time and some sales, so aim it where the answer would move the most money.
  • Decide beforehand what you'll do with the result.Write down the lift that would make you keep, cut or scale the channel before the test begins.
  • Check the test can detect what you're looking for.With too few regions, too short a run or too few sales, a real effect will look like noise.
  • Leave everything else alone while it runs.A promotion, price change or budget shift mid-test blurs the comparison.
  • Retest.A result describes one channel, at one spend level, in one season.
Technical note. Run a power analysis for the minimum detectable effect before committing. For geo tests, matched-market or synthetic-control designs cut noise, and neighbouring regions can leak into each other. A full on/off holdout gives the average incremental return on all of the channel's spend; testing a spend increase or decrease measures the return at the margin.

Mix modelling

  • Ask for ranges as well as the headline figure.A channel shown at a 3.0 return that could plausibly sit anywhere from 1.0 to 5.0 calls for a test before a budget shift.
  • Check that it can predict.Hold back the most recent weeks, have the model forecast them, and compare the forecast with what happened.
  • Calibrate it with test results.A model left alone finds patterns. A model anchored to experiments is tied to something that was measured.
  • Give it variation to learn from.If all your channels rise and fall together, vary spend by channel or region on purpose so the model can separate them.
  • Keep it at channel level.It can tell you how much to put into paid social. It can't tell you which ad to pause.
Technical note. Adstock and saturation assumptions drive ROI estimates as much as the data does, so inspect them. In Bayesian models, lift results can enter as priors on channel effects. Report out-of-sample error and posterior intervals. Geo-level hierarchical models add effective sample size when national history is short. Open-source implementations such as Google's Meridian, Meta's Robyn and PyMC-Marketing make the mechanics inspectable.
Part 4

Using them together

The three methods are weak in different places, so each can check the others. Analysts call this triangulation.

Speed and proof pull in opposite directions

The method that updates fastest gives the weakest evidence of cause, and the method with the strongest evidence is the slowest. No single one gives you both, which is the practical reason to combine them.

What each pair does for the other

calibratecorrectallocateIncrementality testsMix modellingAttribution

Test results anchor the model, and the model shows what to test next.

Incrementality tests to mix modellingA lift result is a measured fact about one channel. Feed it into the model so its estimate for that channel is anchored to something real.

Mix modelling to incrementality testsThe model shows which channels carry the most spend and the widest uncertainty. Those are the ones to test next.

A working rhythm

Every week
Attribution, to steer campaigns and creative inside each channel.
Every month or quarter
Refresh the mix model and review the budget split.
Every quarter
One or two tests, on the channels with the most spend or the most uncertainty.
Every year
Plan next year's budget from the model's scenarios, calibrated by that year's tests.

When the numbers disagree

For questions of cause, a well-run test outranks a model, and a calibrated model outranks attribution. For speed and detail, the order reverses.

Disagreement between methods is information. These are the cases that come up most.

Attribution says retargeting is our best channel. The test found almost no lift.

Believe the test. Retargeting and branded search reach people who were already about to buy, so attribution flatters them. Reduce spend in steps and watch total sales, not attributed sales.

The mix model rates a channel highly. A test of that channel found little.

First check that they measured the same thing. A test captures the effect during the test window at current spend. A model may include slower effects that carry over for weeks. If they still conflict, the test stands for what it measured, and the model should be recalibrated with it.

Technical note. Compare like with like: the model's short-term contribution over the test window against the lift estimate. Check whether the lift's confidence interval overlaps the model's posterior interval before calling it a disagreement.
Attribution shows nothing for video and podcasts. The model says they're working.

This is expected. Attribution can't see channels that people don't click. Before a large increase, confirm with a region test.

Two tests of the same channel gave different answers.

Look at what changed between them: season, creative, spend level, regions. Each test describes its own conditions. If the two ranges overlap, the results may not disagree at all.

Want a second opinion on your setup?

Bconic offers a free 30-minute audit of your current measurement.

Get an audit