How do you measure automation ROI? after go-live, not on a slide
An implementation that works and one nobody uses look identical in a report. Below, measurement treated as a job of its own: what to write down before the change, which metrics you can read out of a system without asking anyone, how to tell a count of operations apart from a real change in the process, and what the 30-day and 90-day readings look like.
- 1 weekhow long the baseline measurement takes before go-live; after the change you cannot reconstruct that number
- 3 to 5metrics written down before the start, each with the system and the filter it is read from
- 83 and 62,985our agent's operations in March and in August 2026, as headcount went from 4 to 23; a measure of adoption, not of effect
- 30 and 90 daystwo readings after go-live: the first one to fix things, the second one to show the result
Short answer
How do you measure automation results so the number holds up?
Measure the process before the change and write the result down. Over a week, not from memory: how many operations a month, how many hours they take, how many come back for rework, how long it takes from event to result. That is the baseline, and it is the one number you can never reconstruct afterwards, because nobody logs manual work.
Pick 3 to 5 metrics you can read out of a system without asking anyone: the automation platform log, timestamps in the CRM, the correction register. Next to each one write the system and the filter. Separately, record how many people actually use the implementation in a week, because without that the rest of the numbers settle nothing.
Then read the same metrics twice: at 30 days, to fix whatever is out of line, and at 90 days, to show the result. Next to the result, list what changed in that window outside the automation. The return in money is a separate calculation, with the formula from the automation ROI guide.
Why does measurement start before the build, not after it?
Once the automation is running, nobody remembers how long the task used to take. They remember that it was „a lot”. A team that has not done the job by hand for a quarter will quote you an old number that moves with how they feel about the change. You cannot reconstruct it backwards.
So measurement is a separate job, and an earlier one. On the process workshop we measure the process for a week, live, with the system log open next to us. A week is enough to catch the exceptions: the Monday pile-up, the Friday close, the one order that always goes a different way.
- Operations per month, counted from a system rather than from memory
- Time per operation, including hunting for the file and asking which version is current
- How many operations come back for rework, and the reason for each one
- Time from event to result, for example from order received to invoice issued
- How many people can do the job when the person who normally does it is on holiday
Which metrics can you read without asking people?
A good metric has one property: a stranger reads it in the same place and gets the same number. A metric that needs a conversation with the team is not worthless, it is an opinion, and the report has to say so.
Below are the metrics we write down most often, with the place each one is read from. The last column matters more than the name: it tells you whether you are measuring execution, effect, or only adoption.
| Metric | Where you read it | What it actually measures |
|---|---|---|
| Operations completed by the automation | execution log in n8n, Make or Zapier | execution |
| Share of runs that finish without an error | the same log, filtered by status | quality of the run, not the effect |
| Time from event to result | timestamps in the CRM or the order system | effect |
| Number of corrections and rework | invoice correction register, document change history | effect |
| People using it in a given week | call log or the list of active accounts | adoption |
| Hours of manual work per task | a weekly measurement, the same one as for the baseline | effect, but it needs a person |
Three to five metrics is enough. With ten, one of them always goes up and the report stops settling anything. If the data sits in several systems, pull it into one place once, BigQuery for instance, and always read it from there. The threshold we hold for error-free runs is 98 percent; below that the scenario goes back for review before anyone starts counting the effect.
What is the difference between a measure of execution and a measure of effect?
The automation completed 12,400 operations last month. It sounds like a result. It is a counter. It says nothing about whether anyone stopped doing the job by hand, whether the process got an hour shorter, whether anyone's week got lighter.
A measure of execution answers how much went through the automation. A measure of effect answers what changed because of it. A report needs both, in that order: the effect first, the execution behind it as the explanation for where it came from.
- Execution: number of operations, share of runs without an error, documents processed
- Effect: time from event to result, number of corrections, hours taken off the team, cases closed the same day
- A pair that works: the automation handled 12,400 orders and time to invoice fell from 26 hours to 3
- A pair that does not: the automation handled 12,400 orders, end of sentence
How do you measure adoption before you start counting the effect?
An implementation nobody uses looks identical to a successful one in a report: scenario green, zero errors, low cost. The difference is that nothing is happening. That is why adoption comes before the effect, as a precondition.
The measure is simple: how many people actually used the tool in a given week and how many operations went through it. If that curve is flat, there is no point looking for an effect. Below is our own log, the only usage numbers we can show without asking a client for permission.
If you have nowhere to read a curve like that, building one is the first job. Platform logs, a list of active accounts and timestamps are enough; how to pull them into a single place is described under AI data analysis.
| Month, 2026 | Agent operations | People using it |
|---|---|---|
| March | 83 | 4 |
| April | 8,029 | 12 |
| May | 24,228 | 22 |
| June | 27,124 | 23 |
| July | 36,585 | 23 |
| August | 62,985 | 23 |
Agent call log in the data warehouse, March to August 2026, counted at the level of a single tool call. This is an agent running in production at a company owned by one of our founders, a marketing agency of around 30 people. CAVEAT: this is a measure of adoption, not proof of effect. It says the tool entered daily work, and nothing beyond that.
What distorts the result and how do you state it honestly?
In the same quarter you switched the automation on, ten other things happened. A season, two people fewer on the team, a new price list, a mailbox migration. Each of them moves the same metrics and none of them has anything to do with the automation.
In a company that is not a laboratory you cannot strip that out. You can describe it, and the description is enough for the report to hold. One rule: the caveat sits next to the number, not in a footnote at the end.
- Seasonality: compare the same months year on year, not this quarter against the previous one
- Team changes: someone leaving or joining moves hours and error counts with no help from the automation
- Other projects in the same window: a new CRM, a data migration, a change in the process next door
- Volume: at twice the number of orders, unit time falls on its own
- A metric redefined mid-measurement, usually by accident, when a report gets repointed at another source
How do you plan the 30-day and 90-day readings?
Two windows, because they measure different things. Thirty days is a check on whether the change took with the team and what still needs fixing. Ninety days is the number for the board. One without the other ends in a report nobody can defend.
Write down the baseline
A week of measurement before the start: operations, hours, corrections, time from event to result. The date and the counting method in the same document.
List the metrics and where each is read
Three to five entries, each with a system and a filter. If nobody but you can read one of them, replace the metric instead of explaining it.
Put both dates in the calendar on go-live day
You book the 30-day and the 90-day reading before production starts. A date set after the fact always lands where the numbers look good.
The 30-day reading: fix things, do not celebrate
The first weeks run low, because part of the team is still doing the job by hand just in case. You are looking for process exceptions and for the places where the automation stops.
The 90-day reading: the result together with its caveats
The same metrics, read the same way, next to a list of everything that changed in that window outside the automation. That is a result. Anything short of it is a claim.
What do you do when the result comes out worse than you assumed?
It happens, and it is not a disaster as long as you say it first. A worse result at 90 days usually has one of four causes, and each one takes two days to check, because all four leave a trace in the data.
What not to do: pick up a new metric that happens to look good, or move the measurement window. The board spots that faster than you think, and at the next implementation they will not believe any number you show them.
- Adoption is flat: check the weekly number of people using it before you touch anything else
- The process had exceptions the workshop did not see: count the operations that come back to a person
- Someone is still doing the same work by hand in parallel: compare the automation log against the operation count in the source system
- The metric measures something other than it was meant to: go back to the baseline and check the definition, it usually drifted when the data source changed
Common mistakes
Six mistakes that leave the measurement proving nothing
Measurement started only after go-live
You reconstruct the old time from the team's memory, and memory moves with how people feel about the change. One sentence takes the result apart, and that is usually how it ends.
A report built out of operation counters
The number of operations says nothing about the process and climbs when a scenario retries the same task after an error. The board asks what it means, and there is no answer.
Adoption left out
An implementation two people out of twenty use looks successful in a report. The effect never arrives and nobody knows why.
A metric that cannot be read without asking people
At 90 days nobody has time to walk around the team. The metric drops out of the report and leaves a hole where the proof was supposed to be.
The measurement window set after the reading
It always lands where the numbers look good. It is visible from outside and it undermines even the parts of the report that were measured properly.
Caveats buried in a footnote
Someone finds them and reads them as concealment. The same sentence, put next to the number, reads as rigour.
Checklist
Measurement checklist to clear before go-live
If any line is a no, the build can start, but you will not be able to defend the result.
- Baseline measured over a week before the start, with the date and the counting method
- Three to five metrics written down before go-live, each with the system and the filter to read it from
- Every metric assigned to a category: execution, effect or adoption
- An adoption measure on the list: people using it in a week and the number of operations
- Both reading dates, at 30 and at 90 days, in the calendar on go-live day
- A running list of events from outside the automation that fall inside the measurement window
- Caveats written next to the number, in the same line of the report
- Agreed in advance what you do if the result comes out worse than assumed
Sources
Sources
- n8n documentation: execution logs, where you read the number of runs and errors
- Make help centre: scenario history and run statuses
- BigQuery documentation: pulling logs and data from several systems into one place to read from
- JustAutomate, AI automation agency: the usage log of our own agent with the measurement method and the caveats
Let us agree the metrics before anything is built
Measurement is planned before the build, not after it
On a free 30-minute consultation we take one of your processes and check what can already be read out of a system today. The process workshop costs EUR 3,500 net per week and ends with a process map, a baseline measured live and a list of metrics you can use even without us. You write directly to Paweł.
Book a free consultationFrequently asked questions
Which metrics should you track after an automation goes live?
Three to five, each read straight out of a system: time from event to result, number of corrections, hours taken off the team, people using it in a week, and the share of runs that finish without an error. Next to each one write down which system it comes from and under what filter.
Why do you measure at 30 and at 90 days?
Because the two readings measure different things. At 30 days the numbers are still low, part of the team is doing the job by hand in parallel, so that reading is for fixes. At 90 days the same metrics give you the result you take to the board.
Is the number of operations completed by an automation a good metric?
It is a measure of execution, not of effect. It tells you how much went through the automation, and it climbs when a scenario retries the same task after an error. Always show it paired with a measure of effect, for example a shorter time from order to issued invoice.
How do you check whether the team is using the implementation at all?
Count people active in a week and the operations going through the tool, from the call log or the list of active accounts. Our own curve went from 83 operations at 4 people in March 2026 to almost 63,000 at 23 people in August. That is a measure of adoption, not proof of effect.
How do you separate the effect of an automation from seasonality?
Compare the same months year on year instead of one quarter against the previous one, and list the changes from outside the automation next to the number: new people, migrations, a new price list. You cannot strip that out in a real company, but a described influence is enough for the report to stand.
What if the result at 90 days is worse than assumed?
Check four things in order: adoption, process exceptions that come back to a person, work still being done by hand in parallel, and the definition of the metric. Each one leaves a trace in the data. Do not add a new metric and do not move the measurement window, because both are visible from outside.
Is measuring results the same as calculating ROI?
No. Measurement answers what changed in the process and rests on metrics read out of a system. ROI turns that change into money and needs two more inputs: the hourly cost of work and the full cost of the implementation. Without honest measurement the payback calculation stays a claim.