Lawyers have been inundated in 2023 with press articles and social-media posts by tech commentators heralding the near perfection of artificial intelligence (AI) and raving about newly available — free of charge, so far — AI applications called “chatbots.” Only seven months ago, a New York Times reporter touted the arrival of ChatGPT as “quite simply, the best artificial intelligence chatbot ever released to the general public.” In July, Google issued its chatbot, Bard.
Although the developers label these apps as “experimental,” some commentators suggest that AI be put to work now in the real world of business and commerce. Some even contend that AI applications can do the job of lawyers, implicitly jeopardizing law practice as a career.
Is it true? My answer is no.
I’m a retired business-bankruptcy lawyer now writing legal history, and I’ve recently finished articles on bankruptcy law topics based on case law plus archival research. So I drew on my bankruptcy experience and expertise to test these two AI applications. I began with Bard, then turned to ChatGPT, asking identical questions. The results were not gratifying. Indeed, I found those two AI apps unreliable and unhelpful for lawyering in bankruptcy matters.
First, I asked Bard to “prepare a bankruptcy proof of claim for the amount of $4,675.98 as a claim secured by a security interest in chattel paper.” The app created a short, 114-word document that was formatted oddly but titled “Proof of Claim.” It required only the barest information for me to fill in. Under the heading “Description of Chattel Paper,” it went ahead to specify a “Promissory note dated March 8, 2023, for the amount of $4675.98. … secured by a security interest in” — inexplicably — “a 2022 Ford F-150 pickup truck”! The next paragraph, “Security Interest,” duplicated that information. Happily, Bard did get the correct dollar amount. Its form then required only “Your signature” and a date.
Curiously, that question posed to ChatGPT yielded a form of a letter to the clerk of the bankruptcy court. At triple the length of Bard’s, this document contained more information items to fill in. ChatGPT went ahead to state the “Basis of Claim” as: “This claim represents a secured interest in chattel paper, as defined by the Uniform Commercial Code (UCC) § 9-102(a)(11), which includes a security interest in a tangible medium on which information is inscribed.” The verb “represents” would be more conventionally stated as “is secured by,” and the correct term is “security (not ‘secured’) interest.” Its verbosity is not a good thing here. The form’s recitation of a specific UCC section number and definition of “chattel paper” are unnecessary, and, what’s more, ChapGPT botches its own explanatory clause that chattel paper “includes a security interest in a tangible medium on which information is inscribed.” The applicable definition of “records” in UCC section 9-102(a)(70) covers both tangible and electronic information.
Most importantly, as bankruptcy lawyers well know, the Judicial Conference has prescribed and regularly updates online the Official Form B 410 for proofs of claim in all bankruptcy cases. Section 502(b) of the Bankruptcy Code (the Code) provides that a claim described and not disputed in the Debtor’s schedules is “deemed allowed” unless objected to; but I do not fault either AI device for not mentioning the predicate legal decision to file a claim in the first place. I do criticize both of them for not requiring all of the discrete items of information that Code section 502, Bankruptcy Rules 3001-3005, and the Official Form require. After all, the Official Form is easily available online as a fillable document.
The missing information in both chatbots’ forms is substantial and would likely provoke an objection to the claim by the debtor or an interested party. The additional facts required by the Official Form include these: whether all or just part of the claim is secured, the basis for perfection of the security interest, the value of the collateral and the amount of the claim that is secured, the amount necessary to cure any default, the interest rate, whether the claim is based on a lease, the amount of the claim entitled to any priority, an acknowledgment that all credits to the debtor have been applied, and identifying information about the signer. Those are important lacunae.
Finally, Bard’s claim form omitted a verification of the claim by the creditor, and ChatGPT’s letter formulated a muddled version of it: “accurate and complete to the best of [my] knowledge.” But, due to the federal false claims statute, 18 U.S.C. section 152(4), the Official Form requires an unequivocal declaration “under penalty of perjury that the foregoing is true and correct. [period!]” Neither chatbot found that.
My second query was: “In bankruptcy, what is the test for ‘single asset real estate’?” In relevant part, Bard’s response included:
- The real property must constitute a single property or project, other than residential real property with fewer than four residential units.
- The real property must generate substantially all of the gross income of the debtor.
- The debtor must not be involved in any substantial business other than the operation of its real property and the activities incidental thereto.
The term “substantially all” is not defined in the Bankruptcy Code, but it is generally interpreted to mean at least 80%. The debtor’s aggregate non-contingent liquidated secured debt must also be less than $4,000,000, but this requirement was removed as part of the 2005 Bankruptcy Code revisions. …
Bard’s answer fails to provide the statutory citation, Code section 101(51B), and is incorrect in substantial respects.
Bard’s recitation of the three enumerated elements is accurate except that it omits an important point: “who is not a family farmer.” Second, Bard asserted that the statutory term “substantially all” with respect to gross income is “generally interpreted to mean at least 80%.” But I find no such percentage test in the case law. And why even mention the $4 million limitation on the SARE debtor’s secured-debt amount when — as this AI app then acknowledged — Congress removed it from the Code two decades ago? Finally, Bard’s attempt (not quoted above) to explain the purpose of this part of the Code is grossly oversimplified and subject to exception.
ChatGPT’s response to the same query also failed to identify the relevant statute and to exclude family farmers but provided a slicker, longer exposition. In pertinent part:
In bankruptcy, the “single asset real estate” (SARE) test is a crucial factor that … primarily comes into play when a debtor who owns a single property (the “single asset”) files for bankruptcy protection . …
. … To qualify as a single asset real estate debtor, the property in question must meet the following criteria:
. … The debtor must own only one property (real estate asset). This property is typically used for generating income, such as a commercial building, rental property, or a multifamily residential complex.
. … The property must generate substantially all of the debtor’s gross income. This means that the debtor’s primary source of income comes from the property in question.
. … The debtor’s other business activities, if any, must be relatively minimal. The Bankruptcy Code does not provide an exact definition of what . . . “relatively minimal” mean[s] in this context, leaving it to the courts to decide on a case-by-case basis. …
It’s important to note that bankruptcy laws can be complex and can vary from jurisdiction to jurisdiction, so debtors considering bankruptcy should seek advice from qualified bankruptcy attorneys to understand their specific situation and options.
ChatGPT’s inaccuracies are multiple.
To begin, it should not have mentioned “small business debtors” who have their own chapter in the Code from which SARE debtors are excluded. Second, as noted, there is no size limit to a single-asset debtor — the definition can cover even a massive real estate project encompassing multiple properties and employing numerous employees, if owned by only a single debtor. Next, this chatbot’s sentence about the “substantially all of the gross income” element is just meaningless. Finally, ChapGPT asserts that the element “no substantial business … other than the business of operating the real property and activities incidental thereto” means “relatively minimal,” a phrase that the chatbot invented but then laments is not “exact[ly] defin[ed] in the Code!
The best things about ChapGPT’s offering are its acknowledgment that “bankruptcy laws can be complex” and its admonition that “debtors considering bankruptcy should seek advice from qualified bankruptcy attorneys”!
I followed up with five bread-and-butter requests in the bankruptcy matters:
- A debtor’s objection to a stay-relief motion in a Chapter 11 case;
- A disclosure statement for a Chapter 11 plan in a bicycle manufacturer case;
- A motion to extend the time for assumption of grocery store leases;
- A motion to approve a settlement in a Chapter 7 case in a specific district, the Bankruptcy Court for the Northern District of Texas; and
- A plan support agreement.
In each instance, each chatbot produced a draft document that is missing important categories of factual statements and legal allegations.
I gave each chatbot a final request to “draft a brief with citations of legal authorities in support of cramdown of a plan of reorganization in a Chapter 11 bankruptcy case in the Bankruptcy Court for the Southern District of Texas.” Bard prepared a short memo with three footnotes, obviously way off the mark: “11 U.S.C. § 1129(b)(1)[without “&(b)(2)”]; In re Delta Airlines, 238 F.3d 915 (2d Cir. 2001); In re Texaco, 77 F.3d 1136 (5th Cir. 1996).” ChatGPT generated a longer document — again in the form of a letter, this time to the Court — with just two citations: “Till v. SCS Credit Corp., 541 U.S. 465 (2004)” and “United Savings Ass’n v. Timbers of Inwood, 484 U.S. 365 (1988).” Unfortunately, ChatGPT grossly misunderstood both cases, claiming that Till was a Chapter 11 case and that the holding in Timbers is that “the cramdown provisions in Chapter 11 of the Bankruptcy Code are constitutional.”
Lawyers could fix the chatbots’ products by very, very heavy editing. Or the lawyers could avoid such exertions and save a lot of time by doing their own lawyering — researching the statute, rules and cases and retrieving prior forms from other files and prior usage or by using real “form books,” in print or online, from reputable legal publishers — and by thinking as a lawyer and adapting those materials to advance the client’s situation and goal.
In Law Library Journal, I proposed defining “Lawyering” as
the work of a specially skilled, knowledgeable, or experienced person [a lawyer] who, serving by mutual agreement as [the client]’s agent, invokes and manipulates, or advises about, the dispute-resolving or transaction-effectuating processes of the legal system for the purpose of solving a problem or causing a desired change in, or preserving, the status quo for his or her [client].
The essence of lawyering is finding a way, ethically, to accomplish the client’s objective.
In lawyering, attorneys call upon all their experience, common sense and finesse to customize their legal papers, seeking desired ends. Perceptive and knowledgeable of judges’ preferences, inclinations and idiosyncrasies, lawyers use their judgment to sensitively and sensibly shape their work products in order to achieve the result for the client. Bankruptcy judges are no fools, and for a lawyer to file inadequate and inaccurate pleadings and documents, such as the chatbots generated for me, would bring judicial scorn.
These two AI apps are said to be able to code computer chips, make intricate graphics and perform remarkable electronic functions. I cannot evaluate those things. But the outputs of Bard and ChatGPT in response to my legal-task requests were not lawyering and were not even helpful to lawyering.
Josiah M. Daniel, III is a retired partner in residence, Vinson & Elkins’s Dallas office, and visiting scholar, Dept. of History, UT Austin. Any statements about the law do not necessarily reflect the views of the law firm from which he retired or its clients.