Disclosure's Black Box: When Decisions Disappear Into the System

For decades, disclosure disputes have revolved around a familiar question: What did the prosecution miss? The Government's recent plans to modernise the criminal disclosure regime suggest that we may soon be asking a different one: What made the system decide it was irrelevant?

The plans follow Jonathan Fisher KC's review of disclosure in the digital age and reflect a reality few criminal practitioners would dispute. A disclosure regime built before smartphones, cloud storage, and the proliferation of digital material is struggling to cope with the realities of modern criminal investigations.

The legal test has not changed. The practical challenges have.

Prosecutors must still review material, determine what supports their case, identify material that may undermine it or assist the defence, and properly schedule and describe unused material. But those tasks are now performed across datasets that may contain millions of records drawn from phones, email accounts, messaging platforms and corporate systems. Disclosure has become an exercise in digital archaeology.

The attraction of artificial intelligence is obvious. Modern review platforms can identify themes, map relationships between individuals, generate summaries, group related documents and surface potentially relevant material at a speed and scale beyond any human review team. Faced with an ever-expanding universe of electronic material, technology appears to offer a lifeboat.

But however sophisticated the technology becomes, it cannot answer the most important disclosure question by itself: what might reasonably be considered capable of undermining the case for the prosecution or assisting the case for the accused?

Disclosure has never been merely about processing information. It is about judgment.

Some of the most significant disclosure failures of recent years did not occur because material could not be found. The problem was that its significance was not recognised.

Consider a fraud investigation involving five million documents. An AI system instructed to identify evidence relating to knowledge of a fraud may successfully locate documents containing terms such as "fraud", "scheme" or "fake". Yet material assisting the defence may consist of a seemingly innocuous email, an overlooked connection between individuals, or a document whose significance only emerges when viewed in the wider context of the case. Finding information and understanding its significance are not the same thing.

That distinction matters. AI does not remove human judgment from disclosure. It embeds it within the system.

In a traditional disclosure exercise, the key decisions are generally made by investigators and prosecutors and can, at least in principle, be examined and challenged. AI-assisted disclosure introduces additional layers of decision-making. Some sit within the choices made by prosecutors and disclosure officers. Others sit within technology that they did not build, do not control and may not fully understand.

The latter may prove the more difficult to scrutinise.

AI systems are built upon proprietary models developed by private organisations. Decisions about training data, model architecture, ranking methodologies, retention policies, licensing restrictions and information-security controls may all influence how a system behaves. Much of that information may be commercially sensitive, protected as intellectual property, and unavailable to those deploying the technology. 

A prosecutor may believe they are asking a system to identify material capable of undermining the prosecution case or assisting the defence. In reality, they are relying upon a technological system that operationalises those instructions through design choices, assumptions and training decisions often made by somebody else.

The disclosure exercise then acquires a further layer of judgment through the way that technology is deployed. Models must be selected. Prompts drafted. Instructions provided. Criteria established for identifying potentially relevant material. Thresholds must be set and outputs tested.

The technology may be conducting the review. Human beings are still shaping the outcome.

Defence lawyers already scrutinise disclosure exercises through disclosure management documents, debates over search methodologies and applications under section 8 of the CPIA. The principle is not new. What may be new is the object of that scrutiny.

Today's disputes often focus on custodians, keyword searches, date ranges and review decisions. Tomorrow's may increasingly focus on the technology itself and the manner in which it was deployed.

• Which model was used?
• How was it trained?
• What types of data and human feedback shaped its outputs?
• What information about its operation was available to the prosecution?
• How was it configured?
• What instructions were provided?
• What validation testing was undertaken?
• What human oversight occurred?
• Was an audit trail retained?

These may sound like technical questions. They are not. They go directly to the reliability of the disclosure exercise itself.

This is where the notion of the "black box" becomes important.

Legal scholar Frank Pasquale warned of the dangers posed by opaque decision-making systems whose workings cannot be meaningfully scrutinised. His concern was not simply that errors might occur. It was that important decisions could increasingly be shaped by processes that were difficult to understand, explain or challenge.

The risk therefore is not necessarily that AI gets things wrong. Human beings manage that perfectly well on their own. The risk is that when it does, nobody can readily explain why.

In most contexts, that may be an issue of governance or accountability. In criminal proceedings, it is potentially a matter of trial fairness. Disclosure depends upon the ability of the defence and the courts to understand and, where necessary, challenge the process by which material has been identified, reviewed and assessed.

That becomes more difficult if the decision-making process itself is only partially visible. AI-assisted disclosure may involve systems whose underlying training methods, design choices and operational logic are not fully accessible to those using them. The difficulty is not merely that the defence may lack access to that information. In some cases, the prosecution itself may not possess it.

Nor is there any guarantee that greater technological sophistication will make scrutiny easier. Recent developments suggest the opposite. As OpenAI's Chief Scientist, Jakub Pachocki, observed last week, "as model capabilities are increasing, monitorability is getting more challenging".

The concern, therefore, is not merely theoretical. In the case before the Wisconsin Supreme Court, State v Loomis, a defendant challenged the use of a proprietary algorithmic tool partly because key aspects of its methodology could not be examined. The court permitted its use, but the case exposed a question that resonates far beyond sentencing: How does a party challenge a decision-making process it cannot fully see? That question may soon lie at the heart of disclosure disputes in criminal proceedings.

There is an irony at the heart of the Government's proposals. They are intended to reduce the burden of disclosure. Yet they may create an entirely new category of dispute. The future battleground may not be the document that was missed. It may be the chain of technological and human decisions that determined it was not worth finding.

The modernisation of disclosure is both necessary and overdue. Given the scale of modern digital evidence, greater use of technology may ultimately prove unavoidable. But the legitimacy of disclosure has never depended on speed. It depends on transparency.

The CPIA may finally be entering the digital age. The challenge for the criminal justice system is ensuring that justice itself does not disappear into a black box.