Last fall, ChatGPT took the world by storm, allowing millions to experiment with “generative artificial intelligence” through a simple and intuitive interface. ChatGPT’s success drew competitors, one of which is called Claude, whose creators claim it can analyze long PDF documents.
To see what Claude could do, I uploaded the petitioners’ and respondents’ briefs from Dobbs v. Jackson Women’s Health, the 2022 case that overruled Roe v. Wade. The surprisingly high quality of the results has significant implications for the review of complex legal documents.
After Claude processed those files, I asked Claude two questions.
The first was: “Please summarize and compare these two court briefs. Describe which one you find more persuasive.”
Apparently Claude leans to the left, as this was the concluding paragraph of its analysis, the full version of which is here:
“Overall, I find the respondent’s brief more persuasive. The brief provides a stronger argument that the right to abortion is protected by the Constitution’s guarantees of liberty and privacy. It makes a compelling case that the viability line thoughtfully balances women’s liberty interests with the state’s interests, and that this standard has proven stable and workable over time. The brief also gives significant weight to reliance interests and the harms to women’s autonomy, equality, and health if abortion is banned before viability. In contrast, the petitioner’s brief does not engage as deeply with the complex individual rights at stake. The respondent offers a more convincing position.”
To be fair, to avoid any issue with Claude’s capacity, I did not upload the petitioner’s reply brief, and that additional brief would likely have affected Claude’s review. The summary was done in under a minute.
I then asked Claude: “Please outline a Supreme Court opinion that would accept the petitioner’s position from those briefs and rule accordingly.” In under a minute, it produced this 1.5-page outline. This is an excerpt from section II of its outline, which was cogent and organized:
A. The viability framework invented in Roe and affirmed in Casey has no basis in the Constitution. It improperly prevents states from regulating abortion as they see fit.
B. Roe and Casey have proven unworkable. The undue burden standard is subjective and impossible for courts to apply in a principled, consistent way.
…
E. Reliance interests do not support retaining these egregiously wrong decisions. Casey’s stare decisis analysis was misguided. Reliance interests are minimal for controversial procedural rulings on deeply divisive social issues.
It’s hard to evaluate Claude’s work on the outline. In places, the outline looks a lot like the table of contents from the petitioners’ brief. And Claude may well have the actual Dobbs opinion somewhere in its memory banks.
But that said, Claude’s outline doesn’t just parrot either the brief or the opinion. Claude clearly did a fair bit of its own analysis.
This experiment taught me two lessons.
First, generative AI technology has far more analytical power about legal issues than I thought. In a matter of seconds, Claude was able to accurately summarize and analyze one of the most complicated constitutional questions of our time.
But second, that power has no obvious guardrails. As Judge Brantley Starr recently noted in his order about the use of AI for legal research, programs like Claude have no ethics or ideology beyond what their programmers may have given them.
I suspect that Claude is entirely “self-taught” and has no particular inclinations beyond what it has “learned” from reviewing millions of documents. (That itself is problematic, because even the vast sample of material that Claude has reviewed will have built-in biases.) And if a future Claude is programmed with a distinct ideological bias, that could be affirmatively dangerous for good decision-making if people blindly rely upon it.
Substantively, Claude has a long way to go before it’s ready to become a law clerk. And problems with bias likely mean that Claude and its successors must be monitored carefully.
But Claude’s technical ability is still remarkable. It devoured two complex appellate briefs in a matter of seconds. Kept within proper boundaries, and with its limitations always kept in mind, generative AI is ready today to assist with the review of the most complicated of legal instruments.