Skip to content
SwankyForge
SwankyForge
All insights
Article9 min read

Do not do ML yet: five signs a rule, a report or a spreadsheet beats a model

Five conditions under which a machine learning project costs more than it returns, what to build instead, and how to tell when a model starts to pay.

Many machine learning projects that disappoint do not fail on the model. They fail before training starts: nobody can say how well the current process works, the decision the model would make is worth a few hundred dollars a month, or the output lands in a dashboard nobody opens. The surveys say the same thing in aggregate. Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. In McKinsey's 2025 survey, 39% of respondents reported any enterprise-level EBIT impact from AI, and most of those put it below 5%. BCG's September 2025 study found 5% of companies getting value from AI at scale and 60% reporting minimal revenue and cost gains despite substantial investment.

We build ML systems for a living, and we will still tell you not to buy one when a rule or a report is cheaper. This note lists the five signs we check for, what each one means for the business, and what to build instead.

The short version for the person signing the budget

What. Five questions about your process, your data and your team. If any answer is "no", a model is probably not the next step.

When. Ask them before a discovery phase, not after a proof of concept. Answering takes an afternoon.

Cost of skipping them. A discovery phase with an outside team typically runs €6k–14k over 2–3 weeks, a proof of concept €14k–38k over 4–7 weeks, and a production system €45k–120k over 10–20 weeks. Those are our published bands; other vendors will differ, but the order of magnitude holds. A rule or a report usually ships in days.

Risk. The expensive outcome is not a model that fails. It is a model that works about as well as a rule you could have written, and now has to be retrained, monitored and paid for every month.

Google's own engineering guidance opens with the same point. Rule #1 of its Rules of Machine Learning is "Don't be afraid to launch a product without machine learning", because basic heuristics often get you most of the way with far less complexity.

Sign 1: you cannot say how well the current process works

What it means. If nobody can state the current stockout rate, the current share of fraudulent orders, or the current time to answer a support ticket, nobody will be able to say whether a model improved it. The project ends with a demo and an argument.

Typical symptom. The goal is phrased as a direction ("reduce stockouts", "personalise the site") rather than a number with a measurement method and a time window.

What to do instead. Build the measurement first. A weekly report of the one number the model would move, computed the same way every week, for four to eight weeks. That report is the baseline any later model has to beat, and it is useful on its own: once the number is visible, it can turn out that a different problem is the one worth solving. Rule #2 in Google's guide says the same thing in different words: design and implement metrics before anything else.

Sign 2: the decision is rare, or the volume is small

What it means. A model earns its cost by making many small decisions slightly better. If the decision happens fifty times a month, the improvement has very little to multiply.

Illustrative arithmetic. Suppose a model improves each decision by 2% of the value at stake.

Decisions per month Value at stake per decision Monthly gain at 2% Months to repay a €14k proof of concept
50 €200 €200 70
200,000 €1 €4,000 3.5

The numbers are illustrative, not measured. The point is the shape: the first row never pays back, even before counting monitoring and retraining; the second pays back within a quarter.

What to do instead. For rare, high-value decisions, give the person making them a better spreadsheet: the five or six inputs they actually use, side by side, with last year's outcome next to each case. A checklist plus a human beats a model trained on fifty examples.

Sign 3: a simple rule already captures most of the value

What it means. Many problems have a rule that handles the bulk of cases. Hold orders above a value threshold that ship to a new address with a new card. Show category bestsellers to visitors without history. Reorder when stock covers fewer than N days of last month's sales. A model has to beat the rule by enough to cover its build and running cost, and often it does not.

Typical symptom. Nobody has tried the rule, so nobody knows how much is left for a model to win.

What to do instead. Ship the rule, measure it with the report from Sign 1, and keep a list of the cases it gets wrong. That list does two jobs: it tells you how much value a model could still add, and it becomes the first labelled data and the test set if you build one later.

When the rule stops being enough. Google's Rule #3 is the counterweight: choose machine learning over a complex heuristic. When the rule has grown into dozens of exceptions that only one person understands, and every new exception breaks an old one, a model is easier to maintain than the rulebook. That is when ML starts to make sense.

Sign 4: your data does not record the outcome you want to predict

What it means. A model learns from examples where the answer is known. If the answer was never written down, there is nothing to learn from.

Typical symptoms.

  • Demand forecasting on sales history that does not record stockouts. When the shelf was empty, sales were zero but demand was not, and the model learns that the product is unpopular on exactly the days it was most wanted.
  • Churn prediction where nobody recorded when a customer left, only that they stopped appearing.
  • Fraud scoring where chargebacks live in a payment provider's portal and were never joined back to orders.

What to do instead. Start logging the outcome now, in the same system as the inputs, with a timestamp. Most outcomes need three to six months of history before a model can be trained and then tested on a later period it has not seen. Run a report or a rule in the meantime. If the outcome arrives late (chargebacks can take weeks), plan for that delay in the design rather than discovering it in production.

Sign 5: nobody owns acting on the output

What it means. A forecast changes nothing until someone orders less stock because of it. A fraud score changes nothing until an order is held. If there is no named person, no system where the output appears, and no agreed action for each value, the model is a cost with no return.

Typical symptom. The answer to "who will use this and what will they do differently on Monday" is a department name, not a person and a decision.

What to do instead. Run the decision by hand for a month with a report. Put the report where the decision is made, and watch whether anyone uses it. If nobody acts on a report, nobody will act on a model, and the problem to fix is the process, not the prediction.

A quick self-check

Answer each question for the process you want to improve.

Question If the answer is no Build this instead Typical effort
Can you state today's value of the metric, and how it is measured? No baseline, no proof of improvement A weekly report of that one metric Days
Does the decision happen at least thousands of times a month, or is each one worth a lot? Too little volume to repay a model A spreadsheet or checklist for the person deciding Days
Have you tried the obvious rule and measured it? You do not know what a model could add The rule, plus a log of the cases it gets wrong Days to two weeks
Is the outcome you want to predict recorded, with a date, for the past several months? Nothing to learn from Outcome logging, then a report Three to six months of logging
Is there a named owner and a defined action for each output? Output will not change anything A manual process run on a report for a month One month

Five "yes" answers do not guarantee a model will pay. One "no" is usually enough to say it will not pay yet.

When ML does become worth it

The same checklist, read the other way, describes a good ML project:

  • The metric has been measured the same way for weeks, and there is a baseline to beat.
  • The decision is frequent, or each one carries enough value that a small improvement matters.
  • A rule exists, has been measured, and has grown too complex to maintain.
  • The outcome has been recorded long enough to train on one period and test on a later one.
  • A named person will act on the output, in a system they already use.

Even then, the first step is small: a simple model compared against the rule on the same metric, first on historical data and then on a share of live traffic. If the model does not beat the rule by more than it costs to run, keep the rule. That is a result, not a failure.

How to check your own case

We run a free six-question teardown at /teardown. It asks about the process, the data and the owner, and one of its possible answers is "do not do ML yet", with what to build instead. If the answer is that a model makes sense, the published bands on /pricing tell you what the next step costs before anyone gets on a call.

Sources

Have a process that costs too much?

Tell us about it. The first call is free, and if AI is not the answer we say so.

Discuss your task