Reason-Giving Without Reasoners? Confronting Generative AI Use in Administrative Processes - Combatting the Code book forum
Frank Pasquale provides the second post in our book forum on Yee-Fui Ng’s Combatting the Code: Regulating Automated Government Decision-Making in Comparative Context. To see all posts, please click here.
Frank Pasquale
25.11.2025
In her Combatting the Code: Regulating Automated Government Decision-Making in Comparative Context, Yee-Fui Ng examines many instances of predictive AI use that raise serious concerns about potential inaccuracy, discrimination, or alienation. Her penultimate chapter, ‘Towards a Framework for Technological Governance’, provides important methods for addressing these problems. The book also articulates normative foundations for a positive ideal of legal decision-making that is enhanced—not replaced by—AI.
This post, an appreciative response to the book, will focus on the reasons given for decisions, particularly given the rise of chatbots that can simulate reasoning processes. My main contention is that the principles animating Combatting the Code should not only lead us to demand reasons for automated decision-making, but in many cases should also require human reasoners to articulate such rationales in response to arguments posed by applicants and litigants.
Narrating Accountability
Accountability is a fundamentally narrative concept. As Ng explains, ‘Apparatuses of the state have to account to the people for their actions and make reparations for any harms suffered’ (14). The idea of accountability as giving an account, or narrative, of how and why a given action occurred, is an immensely fruitful one for those seeking to ensure that state action is reasonable. Lack of explanation is part of a more general ‘crisis of narration’ described by Byung Chul Han. When complex, probabilistic methods are deployed in order to allocate benefits and burdens, citizens may worry that the explanatory chain between acknowledged facts and bureaucratic action is missing or incomplete. Requiring reason-giving for action is one way to address their concerns.
Of course, there are some clear limits on demands for reasons. Some administrative actions are too simple to require explanation. For example, if there is video of a driver passing under a red light, it would be unreasonable to demand a full explanation of why a ticket was written. A simple description of the situation, identified by a computer, suffices. Similarly, the stakes of a decision may be low enough that explanation is more of a luxury than a necessity. Think, for instance, of a person applying for a sport fishing license online. While such leisure activities are very important to those pursuing them, there is no reasonable expectation of instant gratification. As long as there is some opportunity to appeal to a person that is reasonably proximate in space and time to the computer's initial decision, automated application of the law is not illegitimate in sufficiently low-stakes and simple scenarios (LSSS’s).
However, LSSS’s are relatively rare. Most administrative actions demand some rationale at the time they are taken. The need for a rationale has disappointed some advocates of legal automation, since it motivates insertion of a human in the loop, undermining the cost savings promised by AI-driven administration. On the other hand, generative AI may take even the task of justification out of human hands, skilled as it is at simulating the patterns of words associated with past, similar rulings.
Delegating Justification
For example, in the United States (US), Dawn Edelman has reported that the Nevada Department of Employment, Training, and Rehabilitation (DETR) will ‘utilize Vertex AI Studio, a Google cloud service’, in order to ‘expedite the ruling process [with respect to unemployment benefits] from three hours to five minutes’. As Todd Feathers describes,
The tool will generate recommendations based on hearing transcripts and evidentiary documents, supplying its own analysis of whether a person’s unemployment claim should be approved, denied, or modified. At least one human referee will then review each recommendation. . . If the referee agrees with the recommendation, they will sign and issue the decision. If they don’t agree, the referee will revise the document and DETR will investigate the discrepancy.
This plan raises some fascinating and disturbing questions about the relative balance between automated systems and state personnel in deciding contested issues. There are some aspects of Nevada law here which seem relatively straightforward to apply. However, there are also numerous potential judgment calls and ambiguous terms, as well as concerns about mis-transcription of hearings. Will generative AI advance fairness here by holistically synthesising the best of past precedents and present filings? Or will it simply provide plausible rationales which judges or referees rubber-stamp in order to increase their measured productivity?
To be sure, rubber-stamping is a problem of longer vintage than automated decision-making systems. For some legal scholars, it may not be a problem at all in some scenarios. Law professor Adam Samaha has developed ‘explanations and justifications for rubber-stamping beyond self-interested schemes, including designs to achieve decision quality at tolerable cost and second-best adaptations to legal constraints and work overloads’. AI could be the ultimate tool for providing ‘rubber-stamp-able’ recommendations for overworked decisionmakers. Predictive AI can match patterns in present case filings to past filings (and thereby recommend the result for a present case that is most like the results that had been generated for the most similar past cases, on some computational approach to ‘likeness’). Generative AI may also reliably output reasons for such decisions.
Thinking as a Non-Delegable Duty
The question I would like to pursue, in light of insights in Combatting the Code, is whether it is legal and moral for generative AI to make justifications so easy for Nevada referees, or legal decision-makers in general. Ng has argued that:
The US has the deepest emphasis on reason-giving within government. The American doctrine of due process has enabled the highest level of disclosure of the inner workings of automated systems, due to the requirement to justify the outputs of automated decision-making by disclosure of the inputs” (186-87).
But what happens when AI is generating reasons, rather than responsible human personnel? One expert quoted in Ars Technica has worried that the insertion of AI in the benefits determination process is problematic because of irreversible errors. As Ashley Belanger reported with respect to the Nevada program mentioned above:
[I]f employees rush and fail to carefully monitor for hallucinations or mistakes, the backlog of appeals could get worse as claimants protest any errant AI rulings. There's also a chance that AI rulings could "undermine the claimant’s ability to appeal that wrong decision in a civil court case," experts warned, because a "district court cannot substitute its own judgment for the judgment of the appeal referee." That means that a lazy human reviewer could rubber-stamp an AI ruling that a court can't overturn, experts suggested.
The concern here appears to centre on the resolution of factual disputes or credibility determinations. These are matters on which US appeals courts do traditionally defer to district courts, and where there has tended to be deference from courts toward the agencies they are reviewing. Such risks have already been documented. Staff of two US federal judges recently ‘used generative AI to draft error-ridden orders [that] misquoted state law, referenced individuals who didn’t appear in the case and attributed fake quotes to defendants, among other significant inaccuracies’. One potential guardrail may be to require the referee to carefully check any fact asserted, or credibility determination made, by the AI. This is a common feature of ‘human in the loop’ review of automated decision-making.
However, there is another layer of concern here that I have not seen articulated in the already substantial press accounts of Nevada’s plan. This is the role of human thought in understanding and legitimately addressing legal disputes. Some may try to sidestep this concern by conceptualising the AI as a stand-in for a human clerk, writing memos to referees the way a law student or recent law graduate would. And yet this analogy fails on one level, and is irrelevant on another.
First, the analogy fails because, while a clerk would actually apply legal reasoning to facts in the world that the clerk has some (opportunity for) experience of, generative AI is merely a next-token predictor. This matters because a legal opinion is not simply a vector of information transmission. Rather, it is supposed to reflect authentic intellectual and emotional engagement with a case. Tania Sourdin and Richard Cornes have convincingly placed ‘intuition, empathy and compassion’ at the foundation of judicial practice. An AI does not share the type of felt and embodied experience that affords some level of understanding by a decision-maker, of the plight their negative decision could impose upon an unsuccessful applicant or litigant.
Second, the analogy between AI and clerk may well be irrelevant because of legal doctrines requiring an active, engaged mental process by the decisionmaker with respect to the actual filings of the applicant/litigant, and not merely their summary by some other person or entity. A contemporary duty of active consideration in Australia can be traced back to Tickner v Chapman (the Hindmarsh Island Bridge Case). This case concerned, among other issues, the refusal of First Nations women to publicly disclose their grounds for considering a parcel of land to be sacred, in the course of a challenge to a proposed development of the land. In Tickner, the Minister failed to directly consider key representations in the case, as was required by s 10(1)(c) of the Aboriginal and Torres Strait Islander Heritage Protection Act 1984. He instead relied on a summary. Justice Burchett stated the following in elaborating on what a ministerial duty to ‘consider’ meant:
What is it to “consider” material such as a report or representations? In my opinion, the Minister is required to apply his own mind to the issues raised by these documents. To do that, he must obtain an understanding of the facts and circumstances set out in them, and of the contentions they urge based on those facts and circumstances. Although he cannot delegate his function and duty under s 10, he can be assisted in ascertaining the facts and contentions contained in the material. But he must ascertain them. He cannot simply rely on an assessment of their worth made by others. . . It is his task to evaluate them, a task he can only perform after he knows what they actually are. (emphasis added)
The focus on the mind of the Minister is clear here. It is not enough for the minister to rubber stamp a document (composed by assistants or a large language model (LLM)) that contains the justifications for action, even if that is the main artifact later courts access in order to assess the decision, or understand its implications. Justice Kiefel further elaborated:
To “consider” is a word having a definite meaning in the judicial context. The intellectual process preceding the decision of which s 10(1)(c) speaks is not different. It requires that the Minister have regard to what is said in the representations, to bring his mind to bear upon the facts stated in them and the arguments or opinions put forward and appreciate who is making them. From that point the Minister might sift them, attributing whatever weight or persuasive quality is thought appropriate. However, the Minister is required to know what they say. A mere summary of them cannot suffice for this purpose, for the Minister would not then be considering the representations, but someone else's view of them, and the legislation has required him to form his own view upon them. (emphasis added)
Justice Kiefel’s requirement that Ministers ‘appreciate who is making’ arguments is another human-centred requirement of legitimate legal decision-making. Perhaps a sophisticated LLM could be fed extensive descriptions of litigants, and thereby have some textual basis for appreciating who they are. But this is a mere simulation of appreciation, consisting in a prediction of the words that would be utilised by someone who actually appreciated who the litigants are.
Justice Kiefel’s distinction between ‘arguments or opinions’ themselves, and summaries of them, is both illuminating and challenging to administer. It is hard to imagine a busy minister reading all the appendices to a case, or the full text of all cases cited in litigants’ submissions. In Carrascalao v Minister for Immigration and Border Protection, another duty of consideration case, there were over 700 pages of submissions. It seems overly controlling to require the Minister to read every page of them. However, core arguments and supporting points, which are potentially decisive in the case, ought to be carefully considered by the Minister (and not simply dispatched with via a machine or assistant). If the volume of cases is too high to permit this, then perhaps more Ministers are needed, or the Parliament must delegate this now-exclusively-ministerial duty to a larger set of actors.
In the US, the emphasis in similar cases is on explanation, rather than consideration. For example, as Ng observes (quoting the leading case MVMA v. State Farm), the Administrative Procedure Act’s arbitrary and capricious ‘standard requires an agency to “articulate a satisfactory explanation for its action, including a rational connection between the facts found and the choice made”’ (55). The word ‘articulate’ suggests the importance of thought, as opposed to mere provision of rationales.
From Rubber-Stamping to Reason-Giving
To be sure, it may be very difficult to determine, post hoc, whether a decision-maker actually considered the losing side’s arguments, rather than deputised their refutation to a machine. In ‘Rubber Stamps’, Samaha briefly considers the possibility of ‘thoughtfulness audits’ to address such concerns, but expresses reservations:
First . . . we have the conceptual challenge of defining rubber-standing and the empirical challenge of verifying it. Second, even random audits cost something, partly depending on the sensitivity of decision makers to positive or negative feedback. . . Perhaps most important, those with power to impose anti-rubber-stamping mechanisms are not always those calling for the effort. There is only a loose relationship between, say, courts controlling the allocation of formal authority and their ability to implement audits and structures of personnel selection that might reduce rubber-stamping.
Nevertheless, work like Ng’s demonstrates that in the algorithmic age, there are some doctrines in the Anglosphere ready to support challenges to human administrators’ overreliance on technology. For example, she explains that ‘If a human over-relies on technology and does not exercise their own discretion in making an administrative decision, the UK and Australian judicial review doctrine of non-fettering discretion may be applicable, allowing legal challenges of such decisions’ (citing Rebecca Williams’s ‘Rethinking Administrative Law for Algorithmic Decision Making’) (205-06). Ng also acknowledges that ‘[t]his doctrine does not exist in the American context’ (206). This comparative approach affords those concerned about the potential overuse of algorithms a roadmap of potential judicial remedies in some jurisdictions, while highlighting their lack (and the consequent need for legislative action) in others.
By exploring doctrines like non-fettering discretion, which requires some attention to the particularities of individual cases, Ng’s Combatting the Code assists those seeking to maintain a truly human-centred system of administrative law. As generative AI’s ability to simulate legal reasoning improves, it will be increasingly tempting for administrators and judges to simply rubber-stamp its recommendations. But such a system of reasons without human reason-givers would substitute a computational simulacrum of justice for the actual human responsibility and action which legitimates a legal system. Courts should reinvigorate doctrines requiring the active mental consideration of cases by responsible decisionmakers, unfettered by particular tools’ output. This will not only help avoid the predictable alienation that would result from totally automated administration, but would also advance important facets of a positive ideal of the practical judgment necessary to apply law to facts.
Frank Pasquale is Professor of Law at Cornell Tech and Cornell Law School.
Suggested citation: Frank Pasquale, ‘Reason-Giving Without Reasoners? Confronting Generative AI Use in Administrative Processes - Combatting the Code book forum’ (25 November 2025) <https://www.auspublaw.org/blog/2025/11/reason-giving-without-reasoners-confronting-generative-ai-use-in-administrative-processes-combatting-the-code-book-forum>