How to verify AI-generated marketing reports before they reach the board

Because the writing and the arithmetic are different skills. A model produces fluent text whether or not the figures behind it are right. Claude's platform documentation lists techniques for reducing hallucinations, says they do not eliminate them entirely, and tells users to validate critical information, especially for high-stakes decisions¹. That is a model provider describing its own limit.

Hallucinations in marketing reporting take familiar forms: a total that does not match the platform, a metric calculated with an unexpected definition, or a confident story about why sales moved.

The calculation step is one weak point. An analytics vendor that sells its own calculation layer argues that a model doing the arithmetic itself, from shop data without ad spend, can return a plausible number built from incomplete inputs². That is a seller's argument, but the point is practical: find out where the arithmetic happens.

There is also the question of what a model does with your data. On the text-to-SQL enterprise benchmark Spider 2.0, which has 632 real-world workflow tasks, an agent built on o1-preview solved 21.3% of them, against 91.2% on the earlier Spider 1.0³. That concerns models from 2024 and enterprise databases such as BigQuery and Snowflake, not marketing data, so it is not a score for today's tools. It is a reminder that turning a question into a correct query on messy real data is hard.

A marketing agency's write-up of a single hands-on test, Claude connected to one Meta Ads account, reports that the metrics were correct, but the assistant blamed ad fatigue for a campaign that had simply ended⁴. One test on one account proves little. It does match where we would look first.

Start with the headline numbers. For every figure that will be shown to the board, find it in the system it came from.

• Spend. Compare with the ad platform's own spend for the same dates and account.
• Revenue and orders. Compare with the shop for the same period and time zone.
• Customers. If the report splits new and returning, check the split against the shop.

Differences are normal when platforms attribute sales differently from the shop. That is not an AI problem. What matters is that the report says which number is which. If a total does not reconcile and nobody can say why, it does not go in the report.

Example (illustrative): An AI-drafted summary states that last month's revenue was €250,000. The shop shows €240,000 for the same dates. The draft's figure includes a week from the month before, because of a date-range mix-up. The text read fine. Only the reconciliation caught it.

A metric can be correct by one definition and wrong by yours. Write your definitions down once and check the report used them.

Our convention for this site:

• MER (marketing efficiency ratio) is revenue divided by all marketing cost, not only media spend.
• Blended ROAS is revenue divided by media spend only.
• Contribution margin is revenue minus the variable costs of an order, counted before marketing.

Those are our definitions, not a universal standard. Yours may differ. The point is that an assistant asked for "efficiency" or "profit" will choose a definition and may not say which. Ask it to state the formula for each number in the report, then compare with yours.

Our view: a report that cannot state its own formulas is not ready for a board.

"Revenue fell because of creative fatigue." "ROAS rose due to the new audience." These are explanations, and models produce them easily.

Treat every "why" as a hypothesis. For each one, ask:

• What data would show this? Is it in the report?
• What else changed in the same period: promotions, stock, pricing, tracking, seasonality?
• Could the opposite explanation fit the same numbers?

If a cause cannot be tested, label it as a guess. Nothing is wrong with a hypothesis in a board pack. It becomes a problem when it is presented as a finding.

Assistants read content: files, web pages, customer messages, review text. Prompt injection is input that changes an AI model's behaviour in unintended ways. OWASP describes indirect injection as coming from external content such as web pages or files, and notes that it is unclear whether any method fully prevents it⁵.

For a reporting workflow, ask:

• What did the assistant read to build this report? Was any of it written by someone outside the company?
• Was it connected through a protocol such as the Model Context Protocol (MCP), and with read-only access?
• Could the report's conclusion have been steered by content in the inputs?

If an input is untrusted, a person reviews it before the report relies on it. There is no guaranteed fix, so the review is the control.

Use a human in the loop: a person with the authority to approve or reject the output. The sign-off is a short record, not a ceremony. A template:

Add the report version and the data sources it drew on. "Looks right" is not a record. "Totals reconciled to the shop and the ad accounts, definitions checked" is.

Our view: a report is ready when someone has put their name next to what they checked.