Judge Rules DOGE Used ChatGPT Illegally to Cut Grants

A federal judge has ruled that the Department of Government Efficiency (DOGE) violated the Constitution when it used ChatGPT to identify and cancel over $100 million in government grants. US District Judge Colleen McMahon issued a sweeping 143-page decision on Thursday, dismantling DOGE's grant cancellation process and highlighting how the agency's reliance on an AI chatbot to make consequential funding decisions was both legally impermissible and methodologically unsound.

The ruling marks one of the most significant legal repudiations of AI-assisted government decision-making to date, raising urgent questions about when and how public agencies can deploy large language models in processes that affect people's livelihoods and constitutional rights.

Key Takeaways From the Ruling

• Over $100 million in government grants were cancelled based partly on ChatGPT's analysis of whether programs were related to diversity, equity, and inclusion (DEI)
• Judge McMahon found the cancellation process unconstitutional, issuing a 143-page opinion detailing the flaws
• DOGE allegedly used ChatGPT to scan grant descriptions and flag anything it deemed DEI-related — a task the model is not designed or validated to perform reliably
• The ruling raises fundamental questions about AI accountability in government operations
• Grant recipients whose funding was terminated may be entitled to have their grants restored
• The decision could set a legal precedent for how federal agencies use AI tools in administrative processes

How DOGE Deployed ChatGPT to Flag DEI Grants

According to the court's findings, DOGE's process for identifying grants to cancel was alarmingly simplistic. The agency reportedly fed grant descriptions into OpenAI's ChatGPT and asked the model to determine whether the funded programs were related to DEI initiatives. Based on the chatbot's outputs, grants were flagged for termination.

The approach ignored several well-documented limitations of large language models. ChatGPT, like all LLMs, is prone to hallucinations — generating plausible-sounding but factually incorrect outputs. It lacks the ability to understand legal definitions, institutional context, or the nuanced policy frameworks that govern federal grant programs.

Judge McMahon's opinion reportedly highlighted that using a consumer AI chatbot to make binding government decisions — without human oversight, validation processes, or established criteria — represented a fundamental failure of due process. The grants that were cancelled funded a wide range of programs, and the court found that many had little or no genuine connection to DEI as defined by any reasonable standard.

The Constitutional Problems With AI-Driven Government Decisions

The ruling zeroes in on a critical legal issue: due process. When the government takes action that deprives individuals or organizations of property — including grant funding they were awarded — the Constitution requires fair procedures. DOGE's ChatGPT-driven process failed to meet even the most basic standards.

Several constitutional issues emerged in the decision:

• No meaningful human review: Grant cancellations were driven by AI outputs without adequate expert oversight
• No notice or opportunity to respond: Grant recipients were not given a chance to contest the AI's characterization of their programs
• Arbitrary classification: ChatGPT's determinations of what constitutes 'DEI' were inconsistent and unreliable
• No established criteria: DOGE failed to define clear, legally defensible standards for what counted as a DEI-related grant
• Lack of transparency: The prompts used and the model's reasoning were not disclosed to affected parties

This is not the first time AI-assisted government processes have faced legal challenges, but the scale — over $100 million in cancelled funding — and the bluntness of the tool used make this case particularly notable. Unlike specialized AI systems built for regulatory compliance, ChatGPT is a general-purpose chatbot with no training specific to federal grant administration.

Why This Case Matters for the Broader AI Industry

The ruling arrives at a moment when governments worldwide are grappling with how to integrate AI into public administration. The European Union's AI Act, which began taking effect in 2024, specifically classifies government AI systems that affect people's rights as 'high-risk' and imposes strict requirements around transparency, human oversight, and accuracy.

The United States has taken a comparatively lighter regulatory approach. The Biden administration's Executive Order on AI Safety, issued in October 2023, included provisions for responsible government AI use, though many of those provisions were subsequently rolled back. DOGE's ChatGPT debacle illustrates exactly the kind of scenario that AI governance frameworks are designed to prevent.

For the AI industry, the case underscores a growing tension. Companies like OpenAI, Anthropic, and Google have invested heavily in positioning their models as enterprise-ready tools. But there is a vast difference between using ChatGPT to draft emails and using it to make legally binding determinations about $100 million in public funding. The DOGE case could accelerate calls for clearer guardrails around government AI procurement and deployment.

Compared to the structured AI systems used in other government contexts — such as the IRS's fraud detection algorithms or the Social Security Administration's disability determination tools — DOGE's use of an off-the-shelf chatbot appears remarkably unsophisticated.

The 'Dumb' Factor: Technical Failures in DOGE's Approach

Beyond the legal issues, the ruling exposes what many AI practitioners would consider basic technical incompetence. Using ChatGPT for a classification task of this magnitude without fine-tuning, validation, or even a structured prompt engineering framework represents a failure of elementary AI deployment principles.

Prompt design matters enormously in LLM applications. A well-designed system would include detailed definitions, examples of edge cases, confidence scoring, and multiple rounds of validation against human judgments. There is no indication that DOGE implemented any of these safeguards.

The court's characterization of the process as 'dumb' resonates with a broader critique in the AI community. Simply having access to a powerful AI tool does not mean every application of that tool is appropriate. LLMs are probabilistic text generators, not deterministic decision engines. They are designed to produce plausible text, not to render legal or policy judgments.

Experts in AI governance have long warned that the ease of using tools like ChatGPT creates a dangerous temptation for organizations to deploy them in contexts where they are fundamentally unsuitable. DOGE's case is now the most high-profile example of that temptation leading to real legal consequences.

What This Means for AI in Government Going Forward

The immediate practical impact of the ruling is that the cancelled grants may need to be restored, potentially costing the government significant administrative effort and creating uncertainty for affected organizations and their beneficiaries. Some programs may have already been shuttered or reduced in scope during the cancellation period.

More broadly, the decision sends a clear signal to federal agencies: AI tools cannot substitute for constitutionally required processes. Any future use of LLMs in grant review, regulatory enforcement, or benefits administration will likely face heightened legal scrutiny.

For AI developers and vendors, the case highlights the importance of responsible deployment guidelines. OpenAI's own usage policies discourage using ChatGPT for high-stakes decisions without appropriate human oversight. The DOGE case demonstrates what happens when those guidelines are ignored.

Organizations considering AI for government contracts should note several implications:

• Purpose-built AI systems with validated accuracy will be favored over general-purpose chatbots
• Documentation of AI decision-making processes will be essential for legal defensibility
• Human-in-the-loop requirements are likely to become standard for any AI system affecting rights or funding
• Agencies may need to conduct algorithmic impact assessments before deploying AI in consequential decisions

Looking Ahead: Legal Precedent and Policy Ripple Effects

Judge McMahon's 143-page ruling is almost certain to be appealed, which means higher courts may weigh in on the constitutional boundaries of AI-assisted government decision-making. The case could ultimately reach the appellate level, where it would establish binding precedent for how federal agencies can — and cannot — use AI tools.

In Congress, the ruling may fuel ongoing debates about AI regulation. Lawmakers on both sides of the aisle have expressed interest in AI governance, though they differ sharply on approach. This case provides concrete evidence that the absence of clear rules can lead to costly and unconstitutional outcomes.

For the AI industry, the message is nuanced but important. The problem is not that AI was used in government — it is that it was used badly, without safeguards, without expertise, and without respect for the legal framework that governs public administration. The future of AI in government depends on getting this balance right.

The DOGE-ChatGPT ruling will likely be studied for years as a cautionary tale about what happens when powerful technology meets careless implementation. It is a reminder that the most important question about AI is not 'can we use it?' but 'should we use it this way?'