How to read an agency report: what to demand instead of vanity metrics

ROAS is conversion value attributed to ads, divided by ad spend. It measures revenue, not profit. The word "attributed" does the heavy lifting. The platform decides which sales it takes credit for, using its own rules.

So the report's ROAS and your shop's revenue answer different questions. The first says what the platform claims. The second says what you sold. They can both be correct and still disagree. If the gap worries you, reconcile the platform numbers with shop orders first.

An Ask HN thread, about SEO rather than paid media, shows the problem from the buyer's side. The poster could not tell whether more visitors came from the agency's work or from their own PR, and asked for a better view than the vanity metrics in their analytics². The channel is different. The complaint is the same.

There is a second issue. A measurement company, which sells measurement reviews, argues that ROAS is a number the agency has a lot of influence over³. That is a seller's argument, so test it rather than adopting it. A simple test: ask which numbers come from the platforms and which from your own sales.

A vanity metric looks like progress and does not connect to money. The usual suspects:

• Impressions and clicks. They show activity. They do not show sales.
• Platform ROAS on its own. It shows credit, not profit or cause.
• Quality Score. A Google Ads diagnostic. It does not pay anybody's invoice.
• Bounce rate. Check how your analytics tool defines it before you compare it across periods.
• "We will scale". In agency reports, usually shorthand for increase budget or scope. It is not a result.

The point of the list is not that these numbers are wrong. They just do not tell you whether the ads paid.

Example (illustrative): An order sells for €100, with €60 of variable costs. In our convention, that leaves a contribution margin of €40, counted before marketing. A platform ROAS of 2 means €50 of ad spend per €100 of attributed sales. After advertising, each of those orders is €10 short. The ROAS looks healthy, and the orders lose money.

JudeLuxe's list of vanity metrics and what to demand instead puts contribution margin by campaign and a split of brand and non-brand results at the top of the "demand" side¹. JudeLuxe is a Google Ads agency selling that kind of reporting. The two items are still sensible.

Brand and non-brand belong apart. Brand search can look excellent because people who already know you are searching for you. That is cannibalisation: paid activity taking credit for sales that would have arrived by another route, for example brand search ads capturing clicks that would have gone to your organic result. Blended together, brand results flatter everything else.

A one-page structure. The order matters.

1. Decision summary. What to increase, change, stop or keep checking. Put this first.
2. Spend against shop revenue. From your shop, not from the platforms. Add MER (marketing efficiency ratio): total revenue divided by total marketing spend, without splitting credit between channels.
3. Contribution margin. By channel or campaign where possible. Contribution margin is revenue minus the variable costs that grow with every order: cost of goods, shipping, payment fees, returns and discounts. In our convention, it is counted before marketing. After marketing, we call it contribution after marketing. Sources differ on which costs they include, so the report should say which ones it counts.
4. New customers. Count them separately from repeat buyers. New-customer CAC (nCAC) is acquisition spend divided by first-time customers only. It stops repeat buyers from making acquisition look cheaper.
5. Platform numbers next to shop numbers. With the gap explained.
6. Brand and non-brand results, separately.
7. Measured, assumed, unknown. Three labels, used on every important number.
8. Risks and next steps.

That is our view of a useful structure, not a standard.

Every number in a report is one of three things. Measured means someone counted it. Assumed means it follows from a rule or a model. Unknown means nobody can say yet.

A good report labels them. A weak report presents all three the same way.

Incremental ROAS (iROAS) is the clearest example. It is incremental revenue, meaning revenue caused by the ads, divided by ad spend. If a report quotes iROAS, ask how it was measured. If the answer is a test, ask what the test could not show. If the answer is "the platform's model", that is an assumption and should be labelled as one.

Be as fair to the agency as to yourself. Some things cannot be measured at your volume. An honest report says so. If every month's "unknown" column is empty, ask why.

Keep the list short and use it every month.

• What decision does this report support? If the honest answer is "none", ask for the decision before the next one arrives.
• Which numbers come from the platforms, and which from our own sales? If the answer is "all from the platforms", you are reading the platforms' opinion of themselves.
• What did your last report change in how we spend money? If nothing, the report is decoration.
• What is assumed rather than measured? Ask for the label.
• What would make this number wrong? A good answer names a specific risk, such as a tracking change or a promotion.

If the report never changes a decision, change the report. For the finance side, see how to justify marketing spend to CFO.

For ZFD: Our view is that every report should end with one of four words: increase, change, stop or keep checking. It should also say what is known, what is assumed and where the risk is.