The Irony of Automation, From Human Error to AI Error
By Ross Brigoli
For decades, one of the strongest arguments for automation was simple:
Reduce human error.
Humans forget steps. Humans get tired. Humans misread data. Humans make inconsistent decisions. Humans skip controls when under pressure. Humans sometimes do the right thing in the wrong sequence, or the wrong thing with complete confidence. So we automated.
We built workflows, scripts, rules engines, robotic process automation, integration platforms, monitoring systems, approval processes, CI/CD pipelines, and enterprise platforms designed to make work more reliable, repeatable, and scalable. And to be fair, it worked.
Automation removed a lot of manual effort. It improved consistency. It reduced operational risk. It helped organizations scale. It allowed businesses to move away from tribal knowledge, spreadsheets, and manual hand-offs toward more structured and auditable ways of working. But now we are entering a new phase of automation, and the irony is hard to ignore.
We automated to reduce human error.
Now we have to manage AI error.
That shift is more important than it first appears.
Traditional Automation Was Predictable
Most traditional automation is deterministic.
A workflow engine follows a configured process. A script executes a defined sequence of commands. A rules engine evaluates known conditions. A CI/CD pipeline performs a repeatable set of steps. An integration platform moves data from one system to another based on predefined logic.
Given the same input, the same rules, and the same system state, we usually expect the same output. That made traditional automation easier to test, govern, and audit. Not always easy, but at least conceptually clear.
We could ask:
Did the system follow the process correctly?
If it did not, we could usually trace the failure to a missing validation, a bad configuration, a broken integration, a failed job, or a defect in the code. AI changes that model.
AI Automation Is Different
Modern AI systems are probabilistic. They do not simply execute predefined rules. They interpret language, infer intent, summarize information, generate responses, classify context, recommend actions, write code, and increasingly call tools on behalf of users.
That is powerful. But it also introduces a very different kind of risk.
An AI system can produce a useful answer one moment and a subtly wrong answer the next. It can misunderstand intent. It can hallucinate facts. It can overgeneralize from incomplete context. It can confidently explain something that is incorrect. It can make a technically valid recommendation that is commercially, ethically, legally, or operationally wrong.
The uncomfortable part is this:
AI error often does not look like system failure.
A traditional system failure is usually visible. A job fails. An exception is thrown. A validation rule blocks the transaction. A deployment fails. A service becomes unavailable.
AI error can look polished, confident, and reasonable. That makes it harder to detect. And that makes it dangerous.
The Risk Increases When AI Becomes Agentic
When AI is only answering questions, the risk is mostly informational.
A bad answer may mislead a user. A hallucinated summary may create confusion. A poor recommendation may require human correction. That is already a concern.
But the risk profile changes significantly when AI moves from answering questions to taking action. An AI assistant gives you an answer. An AI agent does something.
It can send an email. Update a customer record. Create a ticket. Generate code. Modify a configuration. Trigger a deployment. Approve a request. Query sensitive data. Call an API. Grant access. Execute a command.
At that point, AI error is no longer just a bad response in a chat window. It becomes an operational risk. It becomes a security risk. It becomes a compliance risk. It becomes a business risk. In some cases, it becomes a customer-impacting risk.
This is why I believe we need to stop treating AI automation as simply the next version of workflow automation. It is not. It is a new category of system that requires a new category of controls.
The Question Has Changed
In traditional automation, the key question was often:
Did the system follow the process correctly?
With AI, we also need to ask:
Did the system understand the situation correctly?
That is a much harder question. Because understanding is contextual. The same AI-generated action may be acceptable in one situation and unacceptable in another.
Summarizing an internal document may be low risk. Sending that summary to an external party is a different risk. Updating a customer record is higher risk. Changing production infrastructure is higher again. Granting admin access is a serious security event.
The issue is not whether AI should be allowed to do these things. The issue is whether the right controls exist around the action.
Controlled Autonomy Is the Real Design Pattern
I think many organizations will make one of two mistakes. Some will move too fast and give AI agents broad access to tools, data, and systems without enough control. They will assume that because the AI is useful, it is safe. Others will move too slowly and prevent AI from doing meaningful work because they cannot fully eliminate the risk. Both approaches miss the point.
The answer is not blind trust. The answer is not blanket restriction.
The answer is controlled autonomy.
Let AI move quickly where the risk is low. Add verification where the risk is medium. Require human approval where the impact is high. Block actions that should never be delegated. Make everything observable, auditable, and enforceable.
This is the same pattern we have seen in other areas of enterprise technology.
We did not stop using cloud because cloud introduced new risks. We built cloud governance, landing zones, IAM patterns, policy controls, network boundaries, and observability.
We did not stop using CI/CD because deployments could break production. We built approval gates, automated testing, rollback strategies, environment separation, and release controls.
We should not stop using AI agents because they can make mistakes. But we do need to engineer the control plane properly.
AI Needs a Security and Governance Layer
For AI to become truly enterprise-ready, we need more than prompt guidelines and acceptable-use policies.
We need architectural controls. We need policy engines that define what an AI agent is allowed to do. We need identity and permission boundaries for agents, not just for humans. We need tool-call approval workflows for sensitive actions. We need audit trails that capture prompts, retrieved context, model responses, tool calls, approvals, and execution results. We need verification layers that check whether a proposed AI action matches policy, user intent, and business rules. We need separation between recommendation, decision, and execution. We need runtime monitoring for prompt injection, data leakage, unsafe tool use, hallucinated instructions, and suspicious chains of action.
Most importantly, we need to treat AI security as an architectural concern, not a compliance afterthought.
A simple filter in front of a model is not enough. An enterprise AI security layer needs to understand context. It needs to know whether the AI is reading, reasoning, recommending, or executing. It needs to distinguish between a harmless response and a high-risk action. It needs to inspect tool calls before they happen. It needs to apply policy based on user identity, agent identity, system sensitivity, data classification, business impact, and approval state. It needs to support human-in-the-loop controls where accountability matters.
Not Every AI Error Is Malicious
This is another important point. Not every AI failure is caused by an attacker. Some errors will come from hallucination. Some will come from ambiguous instructions. Some will come from poor retrieval quality. Some will come from stale data. Some will come from missing business context. Some will come from badly designed tools. Some will come from users asking the AI to do things they themselves should not be allowed to do. And yes, some will come from malicious input, prompt injection, or deliberate misuse.
That is why this problem sits at the intersection of security, architecture, governance, risk, compliance, and product design. It is not just a model problem. It is a system design problem.
The Next Enterprise AI Challenge Is Trust
The first phase of AI adoption was excitement. The second phase was experimentation. The third phase is integration. The fourth phase will be risk management. This is where many organizations will either mature or stall. Because without trust, AI adoption eventually hits a ceiling.
People may experiment with AI. Teams may build prototypes. Departments may run pilots. Executives may sponsor initiatives. But for AI to be deeply embedded into business operations, organizations need confidence that it can be controlled. That confidence will not come from better prompts alone. It will come from architecture. It will come from governance. It will come from observability. It will come from well-designed control points between human intent, AI reasoning, and system execution.
The Irony and the Opportunity
The irony of automation is that we spent decades reducing dependency on human judgment because humans make mistakes. Now we are learning that intelligent machines make a different kind of mistake. But this does not mean we should retreat from AI. It means we need to design better systems around it.
The future is not human versus AI. It is human judgment, machine intelligence, and well-designed controls working together. The organizations that get this right will not be the ones that blindly trust AI or completely block it. They will be the ones that learn how to safely delegate work to AI systems while keeping accountability, control, and governance intact. That, to me, is one of the most important architecture challenges of the next few years.
I have been contemplating building an open source AI security layer to solve parts of this problem. Should I do it?
Originally published on LinkedIn