Algorithmic authority

This is article 3 of 5 in The Broken Ballot, a series on AI, deliberation, and responsible corporate governance published every Thursday. Read article 2 here.

What good judgment demands, and what AI can provide

The most clarifying test for any AI governance architecture is to apply to it the best available account of what good judgment actually requires. Sir Andrew Likierman's The Elements of Good Judgment (HBR, January–February 2020) provides exactly that account.

The courts of England and Wales have now, in substance, endorsed this standard for directors. At paragraphs 65–68 of ClientEarth v Shell [2023] EWHC 1137 (Ch), Trower J held that the balancing of multiple competing considerations in corporate management is fundamentally a matter for directors, not the courts – a judicial endorsement of precisely the kind of multi-factor judgment Likierman describes. The court did not address whether the institutional architecture through which shareholders evaluate that judgment is adequate. That is the question this series addresses.

"Those with ambition but no judgment run out of money. Those with charisma but no judgment lead their followers in the wrong direction. Those with drive but no judgment get up very early to do the wrong things."– Sir Andrew Likierman

The values hidden in the code

Every AI governance system embeds normative assumptions in its design. This is not a design flaw, it is an unavoidable feature of any system that operates on the world. But it is a feature that must be confronted honestly.

Consider the discount rate applied to future climate costs in a scenario modelling module. A discount rate is the mechanism used to compare costs that happen at different points in time. The core intuition is straightforward: £1 billion of flood damage to coastal communities in 2056 is not treated the same as £1 billion of damage today. At a 5% discount rate, that future damage is worth only around £230 million in today's terms. At a 1.5% discount rate, the same physical harm, the same number of people affected, the same ecosystems destroyed, is worth around £640 million today. The numbers are different. The damage is identical.

This is not a hypothetical problem. It is precisely the argument that divided the two most consequential climate economists of the last two decades. Nicholas Stern, in the 2006 Stern Review, applied a discount rate of approximately 1.4% and concluded that the costs of climate inaction were enormous, that future generations' interests deserved substantial weight, and that urgent action was economically justified. William Nordhaus, whose approach has underpinned much mainstream climate modelling and who was awarded the Nobel Prize in Economics in 2018, applied a rate closer to 4–5% and concluded that slower, more gradual action was rational. Both economists were working with the same underlying climate science. The difference in their conclusions came almost entirely from a single ethical choice: how much the welfare of people alive in fifty years should count relative to the welfare of people alive today.

The choice between these rates is not a technical parameter, it is a moral and political judgment about intergenerational equity. Yet in an AI governance platform, that choice is typically made once, at system design, by a small team of engineers and product managers, and then embedded invisibly in every governance output the system produces. A shareholder using the platform sees scenario outputs, risk weightings, and probability distributions. They do not see the discount rate buried in the modelling engine that shaped those outputs. A 5% assumption can make a deeply harmful thirty-year project appear financially defensible. The shareholder has no way of knowing that the system has been systematically discounting future harm, because that assumption was fixed in a software meeting they were never party to and cannot scrutinise.

This is what makes the risk so insidious. Democratic deliberation is preserved in form, the vote takes place, the debate appears open, the process is documented. But the range of conclusions available to the deliberating institution has already been quietly constrained by a value judgment made invisible by its expression in code rather than in prose.

The room where the decision is really made

Process architecture, however sophisticated, cannot by itself guarantee good institutional judgment. The most significant governance failures of recent decades have occurred in institutions with formally sound structures. The real mechanism was social, not procedural: the suppression of dissent, deference to dominant figures, the pressure to cohere around a prevailing view, and the marginalisation of independent voices, including those with the most relevant expertise.

Research on board dynamics, including foundational work by Randall Peterson at London Business School, documents how governance failure often originates not in information gaps but in group dynamics deeply embedded in board culture. High-status boards with founder concentration are particularly susceptible. Where the dominant shareholder commands both formal control and informal authority, deliberative processes can be experienced as performative rather than substantive.

Psychological safety, the condition in which individuals feel able to raise concerns, challenge received wisdom, and advocate minority positions without social penalty, is not an aspirational cultural feature. It is a functional precondition for genuine collective judgment.

The AI framework introduces a form of institutionalised dissent: the devil's advocate summary in Stage 4 creates documented engagement with counter-arguments that is harder to suppress informally. The mandatory scenario branching requires the board to engage with adverse scenarios rather than present only the management base case. The post-vote audit trail makes the quality of deliberative engagement legible and auditable.

The future of governance is not fundamentally about AI or ESG disclosure. It is about whether institutions remain capable of genuine fiduciary judgment. It is about whether companies can evolve from value-extraction systems into institutions of responsible judgment. AI is only useful if it helps humans recover what current systems often suppress: attention, responsibility, ethical imagination, and the courage to deliberate before acting.

The power problem: governance capture and the limits of architecture

There is a deeper failure mode that neither process architecture nor cultural accountability can fully address: the capture of governance systems by the actors with the greatest structural interest in their outcomes.

Consider the concentration dynamics already operating in institutional shareholder governance. BlackRock, Vanguard, and State Street, what governance scholars have termed the 'giant three', collectively hold voting rights over a substantial share of most major index constituents. If AI deliberative platforms are adopted at scale, a new concentration risk emerges: which platform dominates becomes a governance question in itself. A world in which the majority of institutional votes passes through a single AI deliberative system is a world in which that system's embedded normative assumptions become the de facto governance standard.

The cultural conditions that architecture cannot supply

Process architecture addresses four of Likierman's six elements with structural mechanisms more reliable than individual will – but cannot manufacture trust or supply experience, both irreducibly human. In an environment where AI systems present outputs with numerical precision carrying unearned authority, where automation bias suppresses questioning, and where deskilling erodes the experiential base on which judgment depends, culture becomes the decisive variable. Kahneman, Sibony and Sunstein (2021) identify noise – random variability in judgments that should be identical – as a second source of error, distinct from bias and requiring different remedies. Both operate at scale when embedded in AI systems. The cultural response must address both.

Epistemic humility as a leadership norm

The most dangerous institutional moment is when false confidence displaces calibrated uncertainty – through individual overconfidence, through hierarchies that reward certainty over accuracy, and now through AI outputs that carry algorithmic authority their accuracy does not always warrant. Epistemic humility as a leadership norm means modelling uncertainty publicly and distinguishing good process from good outcomes. Edmondson's research confirms that psychological safety matters more as uncertainty grows; in AI-augmented institutions, this must explicitly include safety to challenge algorithmic recommendations without personal cost.

Structured dissent as institutional practice

Most institutions claim to value dissent. Very few build it structurally into their decision processes. The difference is decisive: dissent after commitment is recrimination; dissent before commitment is governance. Structural dissent means pre-mortems, devil's advocates, red teams, and the separation of information-gathering from evaluation – which Kahneman et al. identify as the most powerful single noise-reduction intervention available to organisations.

Deliberate experience accumulation, protected from deskilling

Judgment requires the right kind of experience, not merely quantity of it. As AI systems absorb cognitive tasks – scenario generation, pattern recognition, consequence modelling – institutional actors develop less of the experiential base on which future judgment depends. The organisational response is deliberate "manual mode" practice: structured case reviews, analysis of AI errors as learning opportunities, and the HRO practice of treating near-misses not as evidence that the system works but as evidence of where human judgment would be needed if the automated layer failed.

Reflective accountability and the feedback loop

Likierman's "delivery" element – accountability for outcomes – is the element most consistently absent from institutional cultures. Decisions are made; the institution moves on; the lessons of experience remain implicit rather than transferable. In an AI-augmented institution, this failure is compounded: an organisation that does not build systematic post-decision review may be feeding corrupted feedback to the AI systems it relies on, reinforcing the very errors it should be correcting.

What leaders actually do

These conditions require specific and consistent leadership behaviours: publicly questioning AI outputs rather than silently deferring; distinguishing intelligent failure from preventable failure; building cognitive diversity into decision teams; and treating post-decision review as a governance obligation. The deeper implication is that the six-stage architecture proposed in this series is a cultural intervention dressed as process design. Architecture without culture produces compliance without judgment. Culture without architecture produces intention without reliability. The binary ballot was not merely a process failure: it was the institutionalisation of a culture in which structured dissent, deliberate option-generation, and reflective accountability were designed out. What the six-stage framework proposes is designing them back in.

The fourth article tests these arguments against a real decision. INEOS's €4 billion Project One at the Port of Antwerp, currently facing its fourth successive legal challenge, provides a stage-by-stage demonstration of how the AI governance architecture would have restructured the shareholder vote, and what it could and could not have guaranteed.

illuminem Voices is a democratic space presenting the thoughts and opinions of leading Sustainability & Energy Thought Leaders, their opinions do not necessarily represent those of illuminem.

Track the real-world impact behind the sustainability headlines. illuminem's Data Hub™ offers transparent performance data and climate targets of companies driving the transition.