Scholarship

The deployment of machine learning methods and artificial intelligence in the context of legal decision-making will require a thorough look at the concept of variance - be it that original to human decision-makers, or that of automated systems. Human judges are, indeed, noisy in their decisions, a fact that, though deplorable from the viewpoint of individual cases, may have systemic value for the legal framework as a whole: in particular, individual variance in legal cases functions as an information-collection device that propagates, throughout the legal system, a potential lack of fit between the norm and its application, in call for a resolution. Besides, in justice as in machine learning, randomness and variance are increasingly not merely a byproduct but a fundamental aspect that enables these systems to avoid being trapped in sub-optimal configurations. In this context, this Note offers some reflections on whether algorithmic methods should seek to suppress or replicate variance in legal decision-making, and what principles should govern any deployment of deliberate, automated variance.

‘Noise’ is the unjustified and unwanted variance in a set of judgments over comparable issues. Together with bias, Noise is a driver of error in decision-making. As argued by the authors of the bestseller ‘Noise: A Flaw in Human Judgment’, every set of judgments or decisions (in legal proceedings or otherwise) evidence statistical ‘Noise’, and more of it than is commonly believed. Such variance has corrosive, if often concealed, consequences in terms of fairness, efficiency and legitimacy. In this article we demonstrate that there is likely to be substantial Noise in international arbitration proceedings, which is driven by features inherent to the arbitral process (though further features also help mitigate it). We present our Noise Audit and identify examples of Noise in publicly-available awards. We conclude with a number of recommendations to minimize Noise, in order to forge a pathway towards greater consistency in international arbitration.