Another week, another legal tech conference. ‘Tis the season for the next four to six weeks. This week it was Relativity’s turn: its conference, RelFest (formerly known as Relativity Fest), kicked off in Chicago. Relativity is the well-known e-discovery and data management software provider.
I’ve always thought RelFest was a little different than your usual vendor conference. Yes, there are sessions about products. Yes, there are testimonials. And yes, presumably somebody somewhere would like you to buy something.
But RelFest also traditionally includes substantive programs featuring people who actually know what the hell they’re talking about. Things like a judicial panel, an annual relevant case review, and A2J discussions.
The Judicial Panel
This year’s conference began with the judicial panel that Relativity has hosted for 13 years. One look at the panelists tells you this is as far from a sales talk as you can get:
Dr. Victoria McCloud Judge (Ret.)
U.S. Magistrate Judge William Matthewman
Judge Michelle Rick, Michigan Court of Appeals
U.S. Magistrate Judge Allison Goddard
U.S. District Judge Xavier Rodriguez
Serious people who are among the most knowledgeable judges in the country when it comes to AI and e-discovery. The panel talked about several things. Including the judicial use of AI, AI training, the application of privilege to AI prompting and queries, and the use of AI in e-discovery. All important topics.
But the thing that attracted my attention was the judges’ discussion of hallucinations. Because instead of the usual fire and brimstone, the judges offered something that’s been badly missing from the hallucination discussion. Perspective.
The Standard Take
Here is the standard take, mainly by pundits who aren’t practicing or dealing with real-life issues lawyers and judges face. Hallucinations are real. They can’t be avoided. They are proliferating, and it’s all the lawyers’ fault. Lazy lawyers who don’t check citations. They should be severely chastised, fined, disbarred. Tarred and feathered. Disgraced. And a conclusion that perhaps we shouldn’t use AI at all.
The World We Live In
Obviously, lawyers who use AI need to check the cites before using them. Full stop. And sometimes, maybe even often, that takes more time than just not using AI at all. But to conclude it’s just a lazy lawyer problem ignores the world lawyers work in.
The associate who’s pressed for time at 2 a.m. trying to get a brief to the senior partner by 9 a.m. and who needs a case to plug into a string cite. The local counsel who gets a 50-page brief with only hours to spare before filing from a national counsel at an Am Law 50 firm. The client who wants lower bills. Everybody wanting everything yesterday. Or even the judge who relies on their law clerk and doesn’t have the time or energy to check all the cites in the clerk’s memo.
For years, lawyers and judges have relied on delegation and trust that didn’t require or assume that they’d check all the cites in work that came to them. I know. I can assure you that I didn’t check all the cites in a memorandum given to me by an associate I knew, trusted, and worked with for several years. I would read the key case or cases. But the third case in a string of cite for the summary judgment standard? Come on. But today? I would check every one. That behavioral shift takes some getting used to.
And after listening to the judges’ more nuanced view of AI and hallucinations, a world which they actually live in and not just write about, I think they offer some pretty damn good perspective on all this. Two points stood out.
We All Make Mistakes. Count On It
Point one. Judge Matthewman began the discussion with a concept I have not heard before in hallucination discussions: “We all make mistakes.” His point: if, or perhaps when, a lawyer cites a hallucinated case, own it. Admit the mistake, advise all involved, withdraw the pleading, and offer to pay opposing counsel’s costs. Fess up.
It’s also an implicit recognition of the real world we practice in and that hallucinations may in fact occur for a whole host of reasons. I can think back to numerous times before AI when, in the heat of the moment, I may have stretched a holding. Or hit reply all when I shouldn’t have. Or sent a text by mistake. We have all done it.
And when it happened to me, I immediately owned up to it and moved on. It didn’t necessarily mean I would be fined or disbarred. And if I had to confess it in front of a judge, I can damn well guarantee you it would never happen again.
We need more of this attitude and less fire and brimstone. Thanks, Judge, for pointing out that in our profession, we all will make mistakes.
The Frequency Illusion
Point two. Judge Goddard talked about what she called the frequency illusion. While everyone screams constantly about hallucinations, the truth is the number of filings with hallucinated citations compared to the millions of court filings every year is pretty small. Hallucinations are, as she put it, “a numerator without a denominator.” And the judges, to a person, think the number of hallucinations is going to go down. In fact, they think it already has.
That doesn’t mean hallucinations aren’t serious. But seriousness and frequency are different things. A plane crash is serious, but it doesn’t mean all planes are crashing. Seriousness doesn’t mean AI use should be banned or discouraged. It doesn’t mean we need to promulgate arcane requirements about declaring AI was used for something when, in fact, its use is pretty much ubiquitous. It doesn’t mean fueling constant questioning and suspicions about lawyers’ pleadings and credibility. We can do without further erosion of trust.
It’s Not Just About Risks. It’s Also About Benefits
Here’s another thing: given all the publicity and hue and cry about hallucinations, many lawyers have just decided not to use AI at all. Ignoring their obligation under Comment 8 to ABA Model Rule 1.1 to know and understand not only the risks of technology, but also its benefits. Depriving their clients of those benefits and potentially placing them at a disadvantage.
And make no mistake, AI is a tool with real benefits. It can reduce costs at a time when litigation costs are already through the roof. Costs that effectively close the courthouse doors for many. And at a time when litigation drags on and on, AI could be used to streamline the courtroom.
Yet, we insist on holding lawyers to perfection when, pre-AI, they made mistakes as well. Mistakes that, if not ignored, were to some extent tolerated. The case with the wrong page or volume cited. A misquote. And yes, a case cited for a proposition for which it arguably did not stand.
Time for a Deep Breath
It’s about time for some reality when it comes to the “hallucination crisis” and the new so-called plethora of lazy lawyers. So, like the judicial panel, let’s all take a deep breath and quit hyping a problem for hype’s sake.
We all make mistakes. Let’s not confuse the numerator with the denominator. How’s that for a little perspective?
Stephen Embry is a lawyer, speaker, blogger, and writer. He publishes TechLaw Crossroads, a blog devoted to the examination of the tension between technology, the law, and the practice of law.