Can AI be trusted in court? Supreme Court's warning opens debate on regulating legal AI

When the Supreme Court recently discovered that two insolvency tribunals had relied on non-existent case laws while deciding a dispute, it did more than correct a judicial error.

It exposed a growing challenge confronting courts across the world: what happens when artificial intelligence becomes part of legal research, but its mistakes go unnoticed?

The court set aside orders passed by the National Company Law Tribunal (NCLT) and the National Company Law Appellate Tribunal (NCLAT) after finding that they cited judgments that simply did not exist.

The fabricated precedents, likely the result of AI hallucinations, have now become a cautionary tale for India's legal fraternity, raising fresh questions about the unchecked use of generative AI in legal practice.

The ruling comes at a time when lawyers, law firms, and even judicial officers are increasingly experimenting with AI tools to summarise lengthy judgments, draft pleadings, and to speed up legal research. While these tools promise efficiency, they have also demonstrated a well-documented tendency to generate false but convincing information.

The Supreme Court's intervention has shifted the conversation beyond technology. The central question is no longer whether AI makes mistakes: its limitations are already known. The real issue is accountability.

If fabricated citations make their way into court records or judicial orders, who bears responsibility? The lawyer who used the tool, the judge who relied on the research, the tribunal that reproduced it, or the technology company that built the AI model?

For Saakar Yadav, founder of Lexlegis AI, the answer lies in recognising the limits of general-purpose AI systems.

“Every court, law firm, and regulator regardless of jurisdiction, is vulnerable when legal research is entrusted to systems that optimise for plausibility rather than truth. The future of legal AI will not be defined by larger models alone, but by verified legal corpora that keep reasoning grounded in authentic sources of law. In legal practice, there is no room for educated guesses,” Yadav told THE WEEK.

His remarks reflect a growing consensus among legal technology experts that AI should assist legal professionals, not replace the rigorous verification that legal research demands.

Unlike conventional legal databases that index authenticated judgments, generative AI systems are designed to predict the next likely sequence of words.

They are not built to distinguish between an actual Supreme Court precedent and one they have inadvertently invented. That makes human verification indispensable.

The controversy has also highlighted the absence of any formal framework governing AI use in Indian courts.

There are no judicial rules requiring advocates to disclose when AI has been used to prepare pleadings or legal submissions. Nor are there prescribed standards on how AI-generated research should be verified before being cited in court.

Several countries have already begun responding to this challenge.

In the United States, multiple lawyers have faced sanctions after filing briefs containing fictitious AI-generated precedents. Some federal judges now require lawyers to certify that every citation has been independently verified and that no AI-generated content has been filed without human review. Similar discussions are underway in Canada, Australia and the United Kingdom.

India, however, is yet to frame comparable safeguards, even as AI adoption within the legal profession accelerates.

Legal experts say courts may eventually consider practice directions requiring mandatory verification of AI-generated citations, disclosure of AI-assisted drafting in appropriate cases, and regular training for judges and lawyers on the capabilities and limitations of generative AI.

The issue extends beyond courtroom efficiency. Judicial decisions derive their legitimacy from accurate application of the law. If fabricated authorities begin influencing judicial reasoning, even inadvertently, they risk undermining public confidence in the justice system.

The Supreme Court's ruling, therefore, is not merely about one insolvency dispute. It is an early warning about the challenges of integrating artificial intelligence into one of the most credibility-dependent institutions in a democracy.

As AI rapidly becomes an everyday tool for the legal profession, the debate is no longer whether courts should permit its use. The more pressing question is whether India's justice system can afford to embrace AI without first putting in place clear rules on transparency, verification and accountability.