When a lawyer discovers that opposing counsel has cited a non‑existent case generated by a generative artificial‑intelligence tool, the filing must be disclosed.
Human judgment remains the linchpin of AI use in law
At a recent Business Law Section showcase titled “AI in the Trenches and on the Bench,” panelists stressed that AI cannot replace the detailed understanding that attorneys bring to a case. The discussion, moderated by Paulette Rodríguez López of Crowell & Moring, featured Jeffrey Huang of Pilot Company, Alina Lee of Aspire Law, Bradford Newman of Eversheds Sutherland, and the Honorable Richard Platkin of the New York State Supreme Court’s Commercial Division.
Judge Platkin likened poor jury instructions to “bad prompt engineering,” warning that both can produce undesirable results. Huang added that generative AI models are designed to be helpful, often to the point of fabricating citations or facts to satisfy a user’s request. Unlike a human clerk, an AI system rarely says “I don’t know.”
Related: AI Hiring Bias Threatens Employers More Than Expected
A 2024 Stanford study cited during the session found that even premium legal‑AI platforms hallucinate between 17 % and 33 % of the time. Panelists recommended that lawyers explicitly define goals, constraints, and context in their prompts to limit these errors, and they emphasized the need for a human‑in‑the‑loop approach throughout the workflow.
Existing ethics rules already govern AI tools
Newman explained that attorneys do not need a brand‑new set of professional‑responsibility rules to manage AI. The ABA Model Rules already cover competence (Rule 1.1), supervision (Rules 5.1 and 5.3), and candor (Rule 3.3). Competence requires lawyers to understand how the tools they use function, while supervision obliges them to review AI‑generated output as they would a junior associate’s draft.
When a court’s procedural rules demand disclosure, or when a client would reasonably expect to know about AI involvement, attorneys must inform the tribunal of the technology’s use. This aligns with the duty of candor, which seeks to prevent misleading the court.
Related: Accountant Negligence: How To Know If Your Case Is Strong Enough
Courts are currently split on whether attorney‑client privilege and work‑product protection extend to AI‑generated communications. In U.S. v. Heppner, the Southern District of New York held that AI outputs were not protected because the client independently retrieved them, effectively treating the prompts as disclosures to a third party. By contrast, the Eastern District of Michigan in Warner v. Gilbarco, Inc. found the same type of output to be protected work product, viewing the AI platform as a mere tool.
Copyright and pro se challenges add layers of complexity
Lee clarified that AI‑generated content is not automatically copyrightable, drawing a parallel to “monkey selfies,” which courts have ruled cannot be owned.
Judge Platkin warned that an influx of pro se litigants submitting AI‑generated filings poses a practical problem for courts. While some judges may be lenient, others are beginning to sanction vexatious filings that contain fabricated citations. The lack of legal training among self‑represented parties makes it harder to spot hallucinations, raising the risk of wasted judicial resources.
Related: Mishandling gender identity data costs life sciences millions
They must verify every output.
In practice, the balance between leveraging AI’s efficiency and maintaining rigorous human oversight will determine how the profession adapts. The panel’s consensus echoed a simple truth: AI can augment legal work, but it does not replace the responsibility of lawyers, judges, and clients to verify and contextualize each result.
