Like other law firms and other professionals, we're seeing people come to us with things they've been told by ChatGPT, or Claude, or Gemini that are off-base or just flat wrong. That's likely to become more common in the future.
As attorneys, our continuing education classes have been filled with instructions on how to - and how NOT to - use artificial intelligence (AI) and large language models (LLMs) like ChatGPT. And more importantly, as attorneys we are usually able to quickly spot when the result we've been given by these tools is wrong somehow.
Here's an example from a quick interaction I had with popular LLMs this afternoon. I had a question about parole eligibility of a certain crime, so instead of logging into Westlaw and pulling up the statute, I figured I would use it as an opportunity to put Gemini 2.5 through a quick test. I asked a very pointed, direct question about when a person conviction of that particular crime would be parole eligible. The response I got back was very wordy, and very wrong. Gemini said that the person would not be eligible for parole at all because the potential maximum sentence was 10 years, and that under Mississippi law no one is eligible for parole if the sentence is less than 10 years. This is false. Gemini then went on to cite the correct statute (Miss. Code Ann. Section 47-7-3), but then quoted made up language to support its incorrect conclusion that the conviction was not eligible for parole. Another swing and miss.
I then thought I'd ask ChatGPT-4o the same question. Just like Gemini, it cited the correct statute. But then it also completely made up text to an otherwise real subsection and quoted it, saying that people convicted of this crime are not eligible for parole. That's false.
Finally, I decided to give Claude a try. Claude did an even worse job than the other two, saying that Mississippi has "truth in sentencing" laws that prohibit parole in all circumstances. (Not true.) Then, in the same answer, it went on to say that all convictions are parole-eligible at 85% of the full sentence. (We used to have 85% parole eligibility in the past for violent crimes, but that hasn't been the case for awhile now.)
The crime I asked it about is in fact parole eligible under Mississippi law at 25% of the sentence, but anyone asking ChatGPT, Gemini, or Claude those questions would be confused, misled, or both. So the moral of the story is that you can't trust LLMs with legal questions, even now. If you have a legal question, you still need to ask an actual lawyer.