Introduction
The first Generative AI (“GenAI”) chatbots were only released less than two years ago. Since then, the business community has embarked on a profound transformation to leverage these technologies. Business leaders, who may at first have been cautious around the new technology, are now demanding that their organizations rapidly adopt tools powered by GenAI. Recognizing the real risks of using these free chatbots, many organizations adopted policies regarding employee use of GenAI chatbots to restrict their use, and train employees on how to experiment with them safely.

But GenAI is rapidly developing beyond chatbots into complex AI systems. This second wave of GenAI-powered systems can drive significant business decisions, enhancing customer experiences, and streamlining operations. This evolution marks a pivotal shift in how businesses utilize technology, moving from basic automation to deep, strategic integration that touches every aspect of an organization's operations. However, as these technologies become more embedded in the core functions of businesses, they also introduce a myriad of regulatory, ethical, and operational challenges. 

The Limitations of Chatbot Policies
When GenAI chatbots started to become more widely available last year, companies were relatively swift to issue guidelines and policies on their use by employees.¹ These guidelines recognized the potential of GenAI and the need for corporate teams to experiment, while also protecting against various cybersecurity, data privacy, and other concerns associated with their use. However, as AI technologies continue to rapidly evolve, chatbot policies alone may no longer be sufficient to address the wide range of legal and compliance challenges new AI technologies present.

First, the employee use chatbot policies adopted by many companies are narrowly-focused on the just that – the risks presented by using publicly-available chatbots. But this approach may fail to address the breadth of AI technologies now being integrated into corporate strategies.² For example, Gartner now predicts that more than 80% of enterprises will have deployed GenAI powered applications by 2026.³ Indeed, as of October 2023, 55% of organizations are piloting or deploying GenAI systems.⁴ Some of these systems will be developed in-house, including by organizations that previously did little, if any, software development.⁵ Thus, more advanced systems may require more careful evaluation than a single policy directed at employee use can provide.

Further, many employee chatbot policies fail to address the current and emerging regulatory landscape for the use of GenAI. In October of 2022, the White House released its “Blueprint for an AI Bill of Rights” which outlines numerous priorities for regulators in addressing AI systems_.6 The Blueprint was followed by an Executive Order directing a whole-of-government approach to addressing regulation of AI technologies.⁷ Federal agencies have taken note. For example, the Federal Trade Commission (FTC) has indicated that it will aggressively pursue the misuse of AI systems, and has done so, by enforcing its consumer protection laws to prevent a retailer from using AI systems to assist in loss prevention^.8 Numerous other federal agencies have issued guidance indicating they intend to pursue companies putting AI systems into use that may violate federal civil rights laws.⁹ But it is not just the U.S. federal government that is stepping up regulation. The EU is now close to formally adopting its AI Act, a comprehensive set of regulations on AI technologies.¹⁰ Further, California is moving forward with proposed rules for automated decision-making technologies, which include the use of AI.¹¹ 

The Case for AI Governance
Rather than a one-size fits all Generative AI policy, there is a growing need for more robust approaches that account for the combination of more complex GenAI systems and the rapidly evolving regulatory environment. Businesses will benefit from a comprehensive framework for AI governance - processes, policies, and standards that guide the ethical, legal, and effective use of AI within an organization. Such an approach should be sufficiently flexible to address not just the use of chatbots by employees, but how AI systems are adopted within a company, developed, and used to facilitate business operations.¹²  

AI governance encompasses a broad spectrum of considerations, from ethical decision-making and bias mitigation to data privacy and intellectual property rights. It is the structured approach that ensures AI technologies are developed, deployed, and maintained in a manner that is legally compliant, ethically sound, and aligned with business goals and societal values.

The implementation of a comprehensive AI governance program offers numerous benefits. First, AI governance can help manage risk by proactively identifying and mitigating potential risks associated with AI, such as operational vulnerabilities, reputational damage, and legal penalties for non-compliance with applicable regulations. Further, AI governance can help ensure that AI systems operate within ethical boundaries, promoting fairness, transparency, and accountability, thus avoiding biases that could lead to discrimination or other harms. Having an AI governance program can also help build confidence among shareholders, consumers, partners, and regulators by demonstrating a commitment to ethical AI practices, thereby enhancing brand reputation and customer loyalty.

AI Governance in Action - Evaluating a Third-Party Vendor
A clear AI Governance process will benefit a company in a number of instances. One particular example is when a business team wants to adopt an AI technology from a third party vendor. Indeed, startups are building new solutions at a furious pace, driven by an extraordinary growth in venture capital supporting those efforts.¹³ But adopting an AI technology presents different challenges from adopting any other software platform. The key to understanding these differences comes from understanding the basic AI formula:

Data + Model = Result

While AI is a complex technology, the basics for adopting a new third-party AI-powered tool or system generally can be boiled down to this simple formula. AI systems use data (often from a variety of sources), to train an AI model to perform a business function. And every AI use case will involve some combination of the data the AI model is trained on. The most common scenario is that a vendor will offer an AI model to accomplish a result, and needs additional data from your company in order to customize and tailor the AI model for your organization. When adopting AI models from third party vendors in this scenario, an AI Governance team should be established so that each of the pieces of the AI formula are given careful consideration:

Data
Data used to train an AI model can come from anywhere. Often, a third-party vendor will use their own data to train a base model offered as part of their AI solution. But the underlying sources of that data may not be entirely clear While many vendors may demur that as asking for the source of data used to train their base model asks for proprietary information, knowing what data was used to train the model can be essential to understanding the risks posed by adopting it. For example, did the vendor have the authority to use that data? Without knowing the source of the vendor’s data it is hard to complete this due diligence.

For example, if the base model is trained on data from the public web, the data may include unlicensed copyrighted information that could be reproduced in outputs for your business, creating inadvertent copyright infringement risks. If it is from other customers of the vendor, did the vendor have the authority to use that data? For legacy companies with large client bases pre-dating GenAI, their agreements with their customers may not have contemplated the use of customer data for model training. Use of third-party customer data that may breach prior agreements the vendor has with its customers could lead to claims of trade secret misappropriation against its future customer if third-party confidential information ends up in outputs for your business. If the data involves consumers, could outputs inadvertently compromise the privacy rights of those customers?

Similar considerations apply when looking to use your own data with the solution. Do you have the authority to use data you may contribute to the solution for model training or otherwise for use with AI? If the data involves consumer data, is that use consistent with existing privacy policies? And what are the risks that the use of the tool could create new privacy harms?

Model
At the core of any AI solution is a model. The model is the system that generates outputs based on its training data (and potentially other outputs) to generate a document, take an action, or otherwise accomplish a business task. For legal professionals, there are unique aspects of this model that are worth considering. For example, these models can occasionally serve as a direct conduit between their training data and their outputs. Thus, information used in training the model can (and often does) show up verbatim in model outputs. As a result, if confidential information is used to train the model, confidential information can be reproduced in the outputs. Thus, any prospective vendor must provide some guidance on how a company’s data will be used and whether company’s data will be used to train the model. If so, will that trained model be available to other customers or the public? Understanding who operates, owns, and has access to a model or system including your data is critical in adopting these technologies.

Result
Of course, any AI system will produce a result of some kind. That result could be a document produced, a draft email, an action taken, or any of innumerable potential effects. This result is the core of the bargain – will the AI system produce the results that we expect? GenAI systems complicate that analysis because they combine immense potential with occasionally chaotic behavior. For example, GenAI systems can produce inaccurate or incomplete information (sometimes called “hallucinations.”). Further, if the AI system is exposed to the public, it may be abused to produce undesirable effects. For example, an internet user was able to convince a chatbot offered by a local car dealer to sell a brand new vehicle for $1, with the model concluding in his answer that it was “a legally binding offer – no takesie backsies.”¹⁴ 

This leads to numerous considerations in adopting the technology. For example, who bears the risk of inaccurate results? In most software agreements, the vendor bears the risk of malfunctions or undesired operation. But with GenAI systems, it is unclear whether undesired performance is the result of bad programming, or data provided by the customer. As a consequence, many vendors offering GenAI products are unwilling to provide the same broad indemnities available with other software products. 

As another example, who is responsible for compliance with relevant laws and regulations? Answering this question has numerous facets. You as the customer will often be putting the results into practical use, and so will bear the initial burden of regulatory compliance. But this leads to two very practical problems. First, why should the customer have to constantly keep up with laws that apply to a product they’ve purchased? Shouldn’t the vendor have better incentive and expertise to make those judgments? Second, compliance may hinge on specific model configurations or other technical details that the customer does not know, and the vendor will not want to share, making it difficult to determine whether the system is actually compliant. 

Conclusion
As the above example shows, adoption of GenAI technologies presents numerous new challenges, and questions when dealing with third-party vendors. The agreements with third party vendors in the AI space will be more complex than a typical software agreement, and must address a wide array of new challenging issues relating both to the technology and evolving regulatory environment. For these and other circumstances, companies need to look beyond chatbot employee use policies, and should consider devising comprehensive AI governance programs.

At Baker Botts, we have an AI Practice Group ready and willing to help navigate these issues. Our team has a wealth of legal and technology expertise to leverage to assist in setting up effective AI governance programs, and evaluate risks around adoption of new GenAI technologies. 

 

ABOUT BAKER BOTTS L.L.P.
Baker Botts is an international law firm whose lawyers practice throughout a network of offices around the globe. Based on our experience and knowledge of our clients' industries, we are recognized as a leading firm in the energy, technology and life sciences sectors. Since 1840, we have provided creative and effective legal solutions for our clients while demonstrating an unrelenting commitment to excellence. For more information, please visit bakerbotts.com.