California lawmakers have approved legislation that would impose new requirements on attorneys who use generative artificial intelligence, including verifying AI-generated work, disclosing its use in court filings, and protecting confidential client information.
Senate Bill 574, known as the Court A.I. Protection Act, passed the Legislature and was sent to Gov. Gavin Newsom for consideration this week. The measure would prohibit attorneys from turning the practice of law over to artificial intelligence while establishing specific duties for lawyers who use AI as an aid in their work.
The legislation comes as tools such as ChatGPT and Claude become increasingly common in legal research, drafting and document review, while courts confront instances in which attorneys have submitted briefs containing cases or citations generated by AI that did not exist.
Under the final version of SB 574, an attorney using generative AI would remain responsible for the legal work produced with the technology. The bill states directly that lawyers cannot “delegate the practice of law” to a generative AI system.
Lawyers would also be required to take reasonable steps to verify the accuracy of AI output, including case and statutory citations, and correct erroneous or fabricated material before using it. Separately, any attorney responsible for a brief, motion, pleading or other paper filed with a California court would have to personally verify every citation contained in the filing, including citations supplied by generative AI.
That requirement addresses one of the most visible problems to emerge from lawyers’ use of generative AI: hallucinations.
Large language models can produce answers that appear authoritative while inventing cases, quotations, statutes, or other information. Courts around the country have sanctioned attorneys after discovering fictitious legal authorities in filed documents.
SB 574 would place the responsibility squarely on the attorney submitting the document. Reliance on an AI tool would not excuse a false citation.
The bill would also require lawyers to disclose the use of generative AI to a court for documents submitted to the court. Attorneys using AI to create material distributed to the public would have to consider whether to disclose its use.
Client confidentiality is another major part of the legislation.
The final bill would prohibit attorneys from entering confidential, personally identifying, or other nonpublic information into a generative AI system if access to that information is not restricted to the attorney and people authorized by the attorney who are themselves obligated to protect its confidentiality.
The provision covers information such as Social Security numbers, dates of birth, medical and psychiatric records, financial information, account numbers, and addresses or telephone numbers belonging to parties, witnesses, victims, and court personnel.
While the bill does not prohibit attorneys from using AI systems with confidential material, it focuses on whether the system adequately limits access to the information entered.
Privacy and confidentiality have become increasingly important legal questions as lawyers and clients use third-party AI platforms.
Earlier this year, U.S. District Judge Jed Rakoff in New York ruled in United States v. Heppner that a criminal defendant’s self-directed conversations with Anthropic’s Claude were not protected by attorney-client privilege or the work product doctrine. The court concluded, in part, that the defendant disclosed information to a third-party AI platform and lacked a reasonable expectation that the communications would remain confidential.
Although Heppner involved a client rather than a lawyer using AI, the case illustrates concerns about sensitive legal information entered into generative AI platforms.
California has already addressed those risks through professional guidance. The State Bar of California updated its Practical Guidance for the Use of Generative Artificial Intelligence in the Practice of Law in May 2026. The guidance addresses attorney competence, confidentiality, supervision, communications with clients, and a lawyer’s duty of candor to courts.
The California Supreme Court also directed the State Bar in 2025 to consider incorporating AI-related principles into the Rules of Professional Conduct. Proposed changes now under consideration would clarify that attorneys remain responsible for exercising professional judgment over both the information entered into AI systems and the material those systems produce.
SB 574 would move some of those concepts beyond professional guidance by placing specific requirements into California law.
The legislation extends beyond attorneys.
Arbitrators would be prohibited from delegating any portion of their decision-making responsibilities to generative AI. An arbitrator also could not rely on information generated by AI outside the evidentiary record without first disclosing its use to the parties and, when practical, giving them an opportunity to respond.
The bill would additionally require the California Judicial Council to revisit its standards governing judicial use of generative AI as the technology continues to develop.
Sen. Thomas Umberg, a Democrat from Santa Ana and chair of the Senate Judiciary Committee, introduced the legislation. His office said the measure is intended to protect people receiving legal services by establishing rules governing AI use by attorneys and arbitrators.
The Senate concurred in the Assembly’s amendments on August 31 by a 39-0 vote, sending the measure to Newsom.
The governor has not yet signed the bill, so its requirements are not currently law in California.
If enacted, SB 574 could become an important test of how states regulate artificial intelligence without prohibiting lawyers from using the technology altogether. The approach accepts that generative AI will increasingly become part of legal practice while preserving a basic principle of professional responsibility: the lawyer, rather than the software, remains accountable for the work presented to clients and courts.
California’s new AI transparency rules took effect August 2, requiring major generative artificial intelligence companies to give the public tools for identifying images, video, and audio created or altered by their systems.The California AI Transparency Act applies to companies whose generative AI systems receive more than 1 million monthly visitors... Read More »
A federal judge’s ruling in United States v. Heppner has become an early warning for lawyers, clients, and companies using generative artificial intelligence in legal matters: conversations with public AI chatbots may not be protected by the attorney-client privilege or the work-product doctrine.The February 17 decision by U.S. District Judge... Read More »
California has had a storied relationship with arbitration. One of the U.S. Supreme Court’s most significant recent opinions on arbitration, AT & T Mobility v. Concepcion, stemmed from a lawsuit filed in California. The state’s attempts to limit the power of mandatory arbitration agreements in both employment and consumer contracts... Read More »
A New York Senate bill would bar artificial intelligence chatbots from presenting themselves as licensed professionals, such as lawyers, doctors, or therapists, when offering advice online. Supporters say the measure aims to prevent businesses from using AI tools in ways that could mislead consumers seeking professional guidance and potentially expose... Read More »
In the wake of concerns about the lack of oversight about the ethics of some of its member-lawyers, the State Bar of California submitted a change to its Rules of Professional Conduct last year, which the California Supreme Court approved with minor changes on June 21. The new rule puts... Read More »
