Skip to content
SwankyForge
SwankyForge
All insights
Article8 min read

95% of AI pilots show no P&L impact, by one count. Agree the metric before you build the model.

Four studies from 2025 and early 2026 agree that most companies see little measurable return from AI, and the common failure sits on the business side, so the fix is a metric, a baseline and a decision rule written before anything is built.

In July 2025 MIT's NANDA project published a report stating that 95% of organisations were getting zero return from generative AI, despite $30–40 billion of enterprise investment. Fortune put the number in a headline in August 2025, and it travelled from there.

The number is weaker than its fame. It rests on interviews with 52 organisations, a survey of 153 senior leaders collected at four industry conferences, and a review of more than 300 public AI initiatives. Critics called the definition of success too narrow. But three other studies with different methods point in the same direction: most companies that use AI cannot yet find it in their financial results.

Our reading is that most of these pilots were decided before a model was trained. Nobody measured a baseline, nobody agreed on a metric, nobody in the business owned the result, and nobody planned how the output would enter the daily process. Those are business gaps, and closing them costs days of work, not months.

Four studies, one direction

Source Sample What was measured Headline finding
MIT NANDA, July 2025 52 organisations interviewed, 153 leaders surveyed, 300+ public initiatives reviewed, January–June 2025 Self-reported return from generative AI 95% of organisations getting zero return; 5% of integrated pilots extracting millions in value
BCG, September 2025 1,250 senior executives and AI decision makers Value from AI across the company 5% generate substantial value at scale; 60% report minimal revenue and cost gains
McKinsey, November 2025 Global survey of organisations Enterprise-level EBIT impact 88% use AI in at least one function; 39% report any EBIT impact, and most of those put it below 5% of EBIT
NBER working paper, February 2026 Nearly 6,000 executives at firms in four countries Effect on employment and productivity over the past three years 69% of firms use AI; nine in ten report no impact on employment or productivity

The definitions differ. MIT asks whether a deployment produced measurable return, BCG sorts companies into maturity groups, McKinsey asks about EBIT, and the NBER survey asks about productivity and headcount. None of them is a clean measurement of AI's effect. Taken together, they say that adoption is wide and measured return is rare.

How much to trust the 95%

Less than the headlines suggest, and the report says so itself. Its own limitations section states that the figures are "directionally accurate" and come from individual interviews rather than official company reporting, that success definitions vary between organisations, and that a six-month observation window may understate success for complex systems. Success was defined as deployment beyond the pilot phase with measurable KPIs, with return assessed six months after the pilot. For task-specific tools, a successful implementation was one that users or executives described as having a marked and sustained effect on productivity or P&L.

The Marketing AI Institute's critique adds that the definition ignores efficiency gains, retention and pipeline effects, and that 52 interviews cannot carry a claim about the whole market. Both points are fair. Treat the 95% as a loud data point, not as a failure rate.

Two parts of the report hold up better than the headline. The first is its diagnosis. The authors write that the divide does not seem to be driven by model quality or regulation, and that most deployments fail because of brittle workflows, a lack of contextual learning and misalignment with day-to-day operations. The second is what successful buyers did: they judged tools on operational outcomes rather than model benchmarks, and they let budget holders and domain managers pick the problems and lead rollouts.

The report also found that external partnerships reached deployment about 67% of the time, against about 33% for internal builds. We are a vendor, so we have an interest in that figure, and it should be read with the report's own caveat: the numbers are self-reported, and companies that choose partners may differ in other ways from those that build in-house. It is a correlation.

Why pilots stall

In our view, four gaps explain most stalled pilots, and none of them is technical.

  1. No baseline. "Handling time fell 20%" means nothing if nobody measured handling time before the pilot, with the same definition, over a comparable period.
  2. No agreed metric. The data team reports model accuracy, finance asks about cost, operations asks about throughput. Each side picks its number after the fact, and the pilot ends in an argument instead of a decision.
  3. No owner in the business. A pilot run by a central team, with nobody on the business side who controls the budget or the process, has no one to roll it out when it works.
  4. No path into the process. The model works in a sandbox, but nobody changed the screen people use, the order of steps, or the targets they are measured on. Output that arrives outside the workflow gets ignored.

For the budget owner: one page before anything is built

Before any data is moved or any model is chosen, write one page with six lines.

  • Metric. One number the business already cares about, with an exact definition.
  • Baseline. Its current value, over a stated window, computed by the query the pilot will use.
  • Decision rule. Which result means roll out and which means stop.
  • Guardrails. What must not get worse while the metric improves.
  • Owner. The person in the business who will change the process if the pilot works.
  • Decision date. The day the decision is made, whatever the result looks like.

An illustrative example for a support copilot that drafts replies to first-line tickets:

Line Illustrative entry
Metric Median handling time per first-line ticket
Baseline Measured over the four weeks before the pilot, same ticket categories
Decision rule Roll out if the median falls by 10% or more; stop if it falls by less than 3%
Guardrails Reopen rate and customer satisfaction within one point of baseline
Owner Head of support
Decision date Six weeks after launch

If this page takes more than a week to agree, the problem is not ready for AI. Sometimes writing it shows that the metric can be moved with a rule or a report, and then the cheaper answer is the right one.

For engineers: making the metric hold up

Randomise at the unit where the effect happens. For a support copilot, that is the agent or the team, not the ticket, if agents would otherwise learn from the copilot and change how they handle control tickets. For operational processes that cannot be split at the same time, alternate periods with a switchback design. Keep a small holdout after rollout to see whether the effect decays.

Measure the baseline with the pilot's own pipeline. Same query, same filters, same window length. Run an A/A comparison first, splitting traffic with no change applied, to see how much the metric moves on its own. Seasonal metrics need a concurrent control group; a before-and-after comparison will mix the pilot with the calendar.

Size the test before you start. For two groups at 80% power and 5% significance, a common rule of thumb is n per group ≈ 16·σ²/δ², where σ is the standard deviation of the metric and δ is the smallest effect worth detecting. Illustrative numbers: if handling time has a standard deviation of 6 minutes and the effect that matters is 0.6 minutes, n ≈ 16 × 36 / 0.36 = 1,600 tickets per group. A team handling 200 such tickets a week needs 16 weeks to collect 3,200. That is the moment to choose a bigger effect, a longer pilot, or a less noisy metric, for example by adjusting for each agent's pre-period handling time. Handling times are skewed, so compare medians or log-transformed values, or bootstrap the interval.

Put the metric into the contract. The definition, the data source, the query, the window and the threshold go into the statement of work as the acceptance criterion for the milestone. Acceptance then means running the agreed query, not debating whether the system feels useful.

Write the kill criterion down before launch. If the effect on the decision date is below the threshold, the pilot stops. Checking the result every day and stopping when it looks good inflates false positives; if early looks are unavoidable, use a sequential test designed for them.

How we work with this

We agree the metric before any model is trained, and it goes into the contract as the acceptance criterion. Milestones are invoiced only after acceptance, so the metric decides the invoice. One named engineer stays on the project from start to finish, which keeps the definition of the metric from drifting between hand-offs. When the one-page charter shows that a rule or a report would move the metric for less, we say so and recommend against buying ML.

On our rate card, a discovery engagement is €6k–14k over 2–3 weeks, and that is the stage where we write the charter above together with the budget owner. A teardown shows how discovery, proof of concept and production are priced, and the contact page is the way to start one.

Sources

Have a process that costs too much?

Tell us about it. The first call is free, and if AI is not the answer we say so.

Discuss your task