Although AI prompt injection attacks currently appear to be ineffective in court, Spader warned that prompt injections were “not among the dangers we envisioned” when courts first began addressing AI-generated legal filings. In Connecticut, as in many other court systems, officials have primarily focused on AI outputs that damage trust in the courtroom—including hallucinated case law and fabricated citations—rather than malicious inputs such as prompt injections.
Spader said courts will likely need to establish formal rules governing prompt injections, especially as Elliott’s case demonstrates how quickly AI technology and its potential misuse are evolving. Without clear guidelines, lawyers could potentially access their clients’ AI tools and use prompt injections to alter court filings without the clients’ knowledge or consent.
According to Spader, the lessons from Elliott’s failed prompt injection attempt extend “far beyond this incident.”
Elliott appears to have relied on AI to develop his legal case without assistance from a qualified attorney. Spader acknowledged that AI tools are increasingly common among professional lawyers, but he said people with less courtroom experience often use chatbots in ways that can weaken their cases.
Spader explained that litigants frequently build their arguments backward, asking AI chatbots to defend a preferred position instead of seeking accurate information or considering opposing arguments. He described this as “a real danger of technology, and one that judges are seeing a lot today,” because overly agreeable chatbots can reinforce claims even when court rulings contradict them. In Elliott’s case, his determination to defend his position ultimately led him to attempt prompt injections designed to persuade the court to agree with him.
“A discussion that is prompted solely to agree with the author is ultimately disingenuous, even to the author,” Spader said. “Those who use these tools should be asked to test a position as easily as they can advance a position.”
Source: arstechnica.com