There’s a growing idea that AI governance can be automated. It sounds great on the surface. Connect your systems, pull in the data, and get a dashboard that tells you everything is under control. Job done.
The reality is a bit less convenient.
AI governance is not really about what your systems are doing. It is about the decisions people make around those systems, and how those decisions are understood, justified, and evidenced. That distinction matters more than most people realise, because it is exactly where automation falls short.
Modern platforms are very good at collecting signals. You can track model performance, usage patterns, drift, compute, and cost without much effort. That kind of visibility is useful, and in many cases essential. But none of it answers the questions that regulators, auditors, or senior leaders actually care about. Why was this system built in the first place? What risks were identified before it was deployed? What trade-offs were accepted, and who made those calls?
Those are not technical questions. They are judgement calls.
Take something as simple as model accuracy. A system might report that a model is 95 percent accurate. That sounds reassuring, but it does not tell you whether that level of accuracy is acceptable. The answer depends entirely on context. If the model is recommending films, you can probably live with a few mistakes. If it is influencing decisions that affect people’s livelihoods or access to services, the same level of error might be completely unacceptable. There is no metric that can make that call for you. Someone has to decide, and more importantly, be able to explain why that decision was reasonable.
The same problem shows up very clearly when you look at things like Data Protection Impact Assessments. A DPIA is not just a form to fill in. It forces you to think about lawful basis, potential harm, the people affected, and whether your mitigations are proportionate. These are not things a system can determine on its own. They rely on understanding context, interpreting regulation, and making defensible decisions. You can store the outcome of a DPIA in a tool, but you cannot generate a meaningful one automatically and expect it to stand up to scrutiny.
Bias is another area where automation tends to overpromise. There are plenty of tools that can detect bias or produce fairness metrics. They are helpful, but they do not solve the real problem. The difficult part is deciding what fairness actually means in your situation and which trade-offs you are willing to accept. Do you optimise for equal accuracy across different groups, or do you aim for equal outcomes? Both approaches have implications, and there is no universally correct answer. Choosing between them is a governance decision, not something you can delegate to an algorithm.
Even when frameworks talk about human oversight, automation struggles. A system might record that a human reviewed an output, but that does not tell you whether the review was meaningful. It does not tell you if the person had the right expertise, whether they challenged the result, or whether proper escalation happened when something looked wrong. A log entry can confirm that something happened. It cannot tell you how well it was done.
Environmental impact follows a similar pattern. You can estimate energy usage and compute requirements, but governance goes beyond measurement. It is about whether you actively tried to reduce that impact. Did you choose a smaller model where possible? Did you rethink how often the system runs? Did you consider more efficient approaches? Those are conscious decisions. They do not appear in system logs unless someone makes the effort to capture them.
Then there is the broader issue of policies and culture. No integration will tell you whether people actually understand your governance policies or follow them in practice. Most failures in AI governance are not caused by a lack of data. They happen because responsibility is unclear, decisions are not documented, or teams simply do not know what good looks like. Culture plays a huge role here, and culture is not something you can extract from an API.
When regulators get involved, this becomes very obvious. They are not interested in whether your model was running smoothly. They want to see evidence. They will ask for your risk assessments, your mitigation actions, who approved what, and how you monitor things over time. That evidence needs to be clear, structured, and attributable to real people. You cannot piece it together after the fact by looking at system metrics.
This is why the idea of fully automated AI governance does not hold up. Automation can help with data collection, monitoring, and alerting, and those things are valuable. But they only cover a small part of the problem. The majority of governance sits in human judgement, decision making, and accountability.
The more honest way to think about it is this. AI governance cannot be automated because it is fundamentally about people, not systems. What organisations actually need is not a tool that pretends to do it all for them, but one that helps them ask the right questions, capture the right evidence, and understand where they need to improve.
If governance could be automated, accountability would not matter. But when things go wrong, nobody is held responsible except the organisation itself. And that is exactly why this problem cannot be solved by automation alone.