Years ago, I attended the annual meeting of in-house attorneys and general counsels in Silicon Valley. And while privacy regulations are something that are omnipresent in tech these days, especially with the growth of the EU’s General Data Privacy Regulation (“GDPR”) and California’s Consumer Privacy Act (“CCPA”), the conference was on the heels of YouTube being fined a record $170 million for violations of the Child Online Privacy Protection Act (“COPPA”).
So, privacy was a hotter topic than normal with one speaker, who heads privacy for one of the world’s biggest companies, stating her belief that Google’s fine was only the “tip of the iceberg.”
Part of what I enjoy about being a lawyer is passing on knowledge. Clients come in clueless, confused, worried. And, at least when they leave my office, I like to leave them more informed — even if they’re still worried. I even spent a few years as a professor. And when I do teach, I like to use hypotheticals. And one of my favorites was given to me by an AI programmer many years ago:
Imagine a bank in France. The bank uses AI to choose who gets a loan. After 5 years, the AI is so good that only 1 percent of its borrowers end up defaulting. At the same time, because it’s real AI, the bank has no idea how its lending decisions are being made.
An Eastern European man living in Paris keeps getting rejected by the AI. Under GDPR, he demands to know why he is being rejected. Simultaneously, he files suit for ethnic discrimination along with several other Eastern Europeans, who have also been rejected for loans by the same AI.
So, curious as to which path Silicon Valley’s best (or, at least, best paid) attorneys would take, I asked 12 people at the conference my favorite hypothetical.
With the exception of two attorneys, all gave nearly the same answer: Involve the board about data policy. Consult with the data protection officer, a job required by GDPR. In short, kick the ball down the road, whether that is to the board or to a company employee.
For those of us who went to law school, these answers shouldn’t be much of a surprise.
Most law students get weaned on the business judgement rule early on. Deferring decisions “to the board” is a near-automatic response to many risks, since it helps to create a safe harbor from shareholders. Similarly, relying on the DPO’s decision creates an emergency scapegoat, where the company’s officers and counsel can blame the DPO in case of any fallout.
However, for most clients, these solutions are of little practical use.
The risk from violating data privacy legislation isn’t shareholders — it’s fines, PR and, potentially, class actions.
Historically, regulatory violations and fines were relatively easy to include in most company’s cost of business calculations. However, that’s not the case with GDPR, COPPA or other forms of regulation that will grow as information technology increasingly pervades people’s lives.
Policymakers and the public are both more concerned about privacy and how data is used. Or, put simply, YouTube’s $170 million fine was only the beginning of a trend that will grow, especially as the demand for AI increases. The greater the demand for a product, the more demand there will be for the precursors or components needed to build that product.
In this case, the more demand there is for AI, the more demand there will be for the human, or truly “original,” data that LLMs require to produce their outputs.
Additionally, in the case of the AI-hypothetical above, there’s no guarantee that either the board or DPO will be able to anticipate all the risks from their potentially racist AI or data privacy rules.
Data protection, privacy and foreign regulation aren’t exactly fun topics for most boards.
Moreover, discussing AI often requires technical knowledge that even highly involved board members simply may not have.
For boards members without a background in development, the fact that the bank’s AI could be racist may be a surprise, as we normally associate racism with human biases. However, for developers the problem of racist AI is a long-standing one. When Microsoft launched its AI chatbot, Tay, in 2016, it turned into a sex-crazed, Hitler-loving neo-Nazi in 24 hours. For many reasons, racial and other forms of bias continue to pop up in AI, whether that is using AI to screen passports, predict crime or approve loans.
And while a DPO may be more familiar with these issues, the DPO position is brand new, created by government mandate. Accordingly, there’s no guarantee a DPO will understand these risks, much less know how to execute solutions which will allow the bank to continue using its AI without getting sued or fined.
Which brings up the third issue: Stifling innovation.
Even if the risks of the bank’s AI are understood by the board, the DPO and even the company’s lawyer, understanding the risk is not the same as finding a solution. Ultimately, the bank wants to innovate. And AI could be a great technology for better lending.
However, if the bank only focuses on its risks, it may end up frozen in time, relying on a slower, human-based loan approval system with a higher default rate.
In short, the answer provided by most attorneys without a background in development, even ones who work in Silicon Valley, are likely either to be ineffective or to directly stifle innovation. Which is an unfortunate result of most legal education being designed for hyper-specialization, rather than cross-disciplinary innovation.
Moreover, giving such legal advice to tech companies to rely on their specific board or DPO may be directly detrimental to the tech industry in general, since any solution developed by each company will become unique to that company and a trade secret. In turn, creating a simple, easy-to-apply compliance standard will take far longer to spread across the tech industry.
In short, in a world in which conventional legal advice is followed, the only way for information to spread about what works and what does not will be to watch the courts as more companies get fined or sued. Which may be acceptable for larger companies, like YouTube, that can absorb a $170 million fine. And it’s great for litigators but disastrous for most startups, which cannot afford to wait around for case law to slowly emerge.
However, to be very honest, as a lawyer myself, I am sympathetic to the answers proposed by the attorneys I met.
Attorneys — myself included — tend to be risk averse for many, many good reasons. (Among them, the fact that our profession wouldn’t survive very long if it was dominated by risk-seeking, adrenaline junkies who constantly push the envelope.) And malpractice lawsuits are ugly affairs.
Accordingly, if your lawyerly advice to a client amounts to “consult with your board and DPO,” rather than implement a solution yourself, the likelihood of a malpractice suit drops considerably.
Meanwhile, if the client brings you to those long meetings with the board or DPO (which he/she likely will), copies you on their emails, etc., then billables go up. In short, less risk and more upside. And while most lawyers try to put their clients first, financial incentives can still shape behavior.
However, as a startup founder who once had to rely on law firms, I know clients want solutions not just advice. Not meetings, consultations, or long research memos.
Moreover, unlike the old days (i.e., pre-1995), when attorneys were gatekeepers of knowledge, the internet has erased much of the legal profession’s competitive advantage. Anyone can look up GDPR or CCPA. Years in law school increasingly does not impart the advantage it once did.
In short, if you are a client paying $1,200 an hour, and your attorney’s advice amounts to “talk to someone in your organization” (i.e. a board member or your DPO), you may be fairly unhappy.
Of the attorneys I spoke to, only two hinted at the same answer I’d come up with from my days working in development. (Years ago, I used to make and publish video games and, before that, built a badly coded reservation platform.) Specifically, “privacy by design.”
When I consider the AI-hypothetical purely from a non-legal, software development viewpoint, the first question I’d ask is how a system is being engineered.
Specifically, 1) how is the product built and checking itself for errors or unacceptable output, 2) what does its documentation look like, and 3) was there external validation and testing.
For instance, there’s the code behind the AI itself. Are there parameters or data related to ethnicity that the AI is prohibited from obtaining?
As advanced as AI is becoming, you still often have to choose which data sets the AI looks through. If the bank designs its AI to omit data sets on ethnicity, the likelihood that ethnicity is being used to grant loans goes down.
Limiting the data sets, or amount of data, that an AI is allowed to analyze may seem to fly against the bank’s commercial interest. However, it may actually be to its benefit.
According to my friends in the entertainment industry, Netflix’s data science team, for instance, does not obtain demographic info on its viewers. And, indeed, Netflix’s own article on its recommendation system does not include any mention of demographic info (i.e., race, age, etc.) being used (https://help.netflix.com/en/node/100639).
Unlike the market research techniques used by the Hollywood studios of old, where race, age and gender were all compiled with painstaking detail, Netflix has presumably determined its resources are better allocated gathering behavioral data like when a subscriber stops watching a show, or gives it a “like,” than how much melanin they have in their skin.
Like humans, AI works by looking at data. The more data there is, the longer it takes to learn. Or, in data science terms, ethnicity produces significant “noise,” which has less predictive power than historical behavior and takes far longer to sort through.
And while the studios of old may have relied on categories such as ethnicity as rough proxies for preference (i.e. producing movies “for minority” audiences or “for women”), in today’s digital age there’s simply no reason to continue to use such proxies.
Today, most preferences can be observed directly through social media posts and a billion other data sources. There are guys who watch shows about baking and glassblowing. Women who watch shows about fast cars. People of every race who watch documentaries about tiger zoos. People no longer need to be lumped together based on traits like race or gender as proxies for predicting behavior.
Were Netflix’s data scientists forced to analyze a bunch of demographic data, on top of all their behavioral data, Netflix may not have over 275 million users. Bad data in = bad data out. And ethnicity is frequently bad data.
The second method for solving the bank’s problem is automating documentation.
Documentation is when a programmer writes notes about what each section of her code does. In short, through documentation, we know why software did something, like reject a home loan.
Without documentation, the economic value of most software falls towards zero, as it becomes impossible for future programmers to find, fix or upgrade the code. (An analogy would be to imagine a library, where you need to find the Odyssey without a filing system or book titles.)
AI decision-making documentation is hard because it’s, in a sense, partly writing its own code. Which may represent one of the last barriers for AI to reach commercial ubiquity. The greater the resistance to documentation, the greater the upgrade and reprogramming/training cost. However, the technology on this issue is now changing rapidly, which will in turn reduce costs.
MIT is developing an AI that can explain its decisions to humans. Other researchers are as well. And while the technology is still in early stages, it will get better over time. Especially when, without getting technical, complying with GDPR’s right to know may be easier than expected, since only broad categories (e.g. spending patterns, debt ratios, social media posts, etc.) and not the weights, are presumably what needs to be disclosed.
The third method is third party validation. For example, build an “evil” AI that seeks to reject loans purely based on ethnicity to compare to the rejection list provided by the bank’s AI. Results can be put on a credit score-like scale, measuring the likelihood that a given AI is “evil.”
This process can be standardized quickly. And clients can use this process to gain a public, external certification that their AI isn’t a Hitler-loving jerk. Which, among other benefits, means they can group themselves together with other ethically minded companies, thereby 1) providing the public some degree of assurance about how their data is being processed, and 2) making it harder for regulators to single out any one company for punishment.
Finally, in addition to limiting datasets, automating documentation, and using third-party validation, newer and emerging techniques like SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), fairness algorithms, and adversarial testing, may also offer additional ways to tackle the challenges of AI bias and transparency.
For example, SHAP and LIME are designed to make AI decisions easier to understand. SHAP calculates how much each factor—like income or credit history—affects an individual decision, while LIME builds a simple explanation around a specific prediction. These tools help companies and regulators see if something like ethnicity is unintentionally influencing an AI’s choices, even when that data isn’t explicitly included.
Fairness algorithms and adversarial testing go a step further by addressing hidden biases and making models more equitable. Fairness algorithms adjust either the training data, the AI’s logic, or its outputs to reduce unfair treatment of certain groups. Adversarial testing works like a stress test, introducing tricky or biased scenarios to identify weak spots in the system. Together, these methods can complement approaches like third-party validation to provide additional layers of accountability.
However, these newer tools aren’t perfect. They can be computationally expensive and hard to implement, especially for smaller companies or startups. Additionally, while SHAP and LIME explain decisions, they don’t necessarily fix biased outputs. Similarly, fairness algorithms and adversarial testing often require tough trade-offs, such as balancing fairness with accuracy or speed, and may produce even larger problems in the longer term by allowing programmers to insert their own subjective biases and beliefs into their models.
Nonetheless, as AI expands, it will increasingly become necessary for attorneys and their clients to understand exactly what tools are available to them to improve their models and avoid an increasing number of ethical and legal pitfalls.
Ryan Riegg is an international tech lawyer at Dallas-based law firm Platt Richmond. He has been published in a few law journals (UCLA Law, Cornell Law, University of Denver/Sturm) and has been featured in Newsweek, Vice, The Hill, and Vox.