An automation that reports success has told you one thing: it ran. It has not told you whether it did the job. The only way to know that is to count what came out against what went in, and to take the "what went in" from somewhere other than the automation itself.
I have seen this failure on two very different teams, with two very different tools, and it looked the same both times. Nothing was broken in the way people expect broken things to look. There were no error emails, no red badges, no alerts. The status said fine. The work was not getting done.
In one back office: "synced" meant nothing
A team rolled out a new timekeeping app. Staff clocked in and out on their phones, and when they clocked out, the app showed a friendly confirmation: synced. From the staff side, the job was done.
From the back office side, a pile was growing. Shifts were arriving with a start time and no end time. Each one blocked the record from going through, and each one needed a person to chase it.
The first response was the natural one, and it was wrong. Every missing clock-out was treated as one person's mistake. Someone forgot to tap the button, so someone gets an email. One record, one email, one reply, one fix. It felt like progress because individual items kept closing.
What it hid was the pattern. Some names came back again and again, from people who were sure they had clocked out and who had a screen telling them so. The app said synced. The record said otherwise. Both could not be true, and only one of them was the record the team actually worked from.
The change was small. Instead of working the exceptions one at a time, I started counting them daily, by date, and keeping a running list of repeat names. Within a short stretch the count stopped looking like scattered forgetfulness and started looking like a trend. A trend with names and dates attached is something you can report upward. A pile of individual chasers is not.
The app's confirmation described the app. The missing end times described the outcome. Only the second one was evidence.
On another team: green runs, empty boards
Somewhere else entirely, a team had an automation that turned client meetings into tasks. A call happened, the automation picked it up, and action items were supposed to appear on the right project board. It had been live for weeks. The dashboard showed runs completing, every one of them green.
Nobody had counted. So I did the simplest check I could think of: list every client call from the calendar for one week, then look for the tasks each one should have produced.
Roughly three of every four client calls had produced nothing. Not wrong tasks. No tasks. And the misses were not spread evenly. Whole types of calls produced zero: calls with new accounts, informal check-ins, anything that did not already have a home.
The rule behind it took a few minutes to find once the count existed. The automation only acted when a project board for that client already existed. No board, no action, quietly, and the run was still reported as complete. From its own point of view it had not failed. It had correctly done nothing.
Why the status light lies
It is not really lying. It is answering a narrower question than the one you are asking.
A status light tells you whether the automation ran to the end without throwing an error. You want to know whether the work got done. Those overlap most of the time, which is exactly why the gap is so hard to see. An automation can run perfectly and skip most of its inputs, because skipping is a valid path through its own logic. No board, no action. No match, no update. Nothing to send, so "synced".
Every automation has a condition like that somewhere: something it checks first, and quietly declines to act on when the check fails. The status light will never show you those declines, because from the tool's point of view they are not errors.
There is a second trap. When you go looking, the obvious place to look is the automation's own log. But the log only lists what the automation saw. If an input never reached it, or reached it and was filtered out early, the log is clean. You have to count from the source: the calendar, the roster, the inbox, wherever the work actually starts.
Count from the source, never from the automation's own log. The log can only tell you about the things the automation noticed.
The check that catches it
You do not need a monitoring platform for this. You need one number, checked daily, and one honest count, done once.
Pick one downstream count
For any automation you rely on, choose a number that rises when it is quietly failing. Records missing a field they should have. Items stuck in a status they should have left. Client calls this week with no tasks. Check it every day, at the same time, and write it down.
The value is not today's number. It is the direction. A count that sits flat is fine. A count that climbs is your alarm, and it will go off days before anyone files a complaint.
Do the input count once
Then, once, do the full count:
- Pick a fixed window. One or two weeks is enough.
- List every input from the source system, not from the automation.
- For each input, tick whether the output a person actually uses exists.
- Tag each input by type, and work out the miss rate per type, not just overall.
- For each type with misses, write the one rule that would explain it, then test that rule against two misses and one success.
The per-type split is the part people skip and the part that matters most. An overall miss rate tells you there is a problem. A miss rate by type tells you where the rule is. In the meeting example the overall number was alarming, but it was the split that pointed straight at "only acts when a board already exists".
Some inputs genuinely need no output. An internal call where nothing was decided is a fair example. Mark those as "none needed" rather than leaving them blank, or you will either inflate the gap or end up arguing about it.
Cover by hand, with an end date
Until the fix ships, someone covers the gap by hand with a short daily check. That is fine, as long as it has an end condition: the fix is confirmed on a fresh window. Manual cover with no end date quietly becomes a permanent job nobody chose. And before creating anything by hand, check whether the automation already created it, so you are not cleaning up duplicates next week.
What to send the owner
Whoever owns the automation does not need a story. They need three things: the rule you think is causing the gap, the rows that show it, and the fix you are asking for. That is a short message, and it is far easier to act on than a run of "this call did not get tasks either" complaints, each fixed on its own while the pattern stays hidden.
I built a small pipeline gap auditor for exactly this count. You list the inputs, their types and their stages, and it gives you the miss rate per type and the stage where each type breaks.
The takeaway
A success light tells you the automation ran. It does not tell you the work got done. Pick one downstream count and check it daily, so a rising number warns you before a complaint does.
Then pick one automation you trust. List last week's inputs from the source, and tick off which ones produced an output. If you cannot do that count, you do not know whether it works. You only know that it runs.
Green means it ran. Only a count of outputs against inputs tells you it worked.