“ChatGPT write me an article on how to govern AI.”
“Sure! Here is a draft article on how to govern AI.”
Full disclaimer: that is not how I wrote this article, but ChatGPT did help me come up with a clever title. Naturally, that piqued my curiosity. What would ChatGPT suggest about how to govern itself? I asked and it was happy to offer several recommendations.
ChatGPT’s first suggestion? Define clear objectives.
Groundbreaking, right?
Let’s unpack that. What does it really mean to “govern AI?” On occasion, a discussion of AI governance will involve mention of guardrails, as if guardrails and governance are synonymous. In the real world, guardrails are designed to keep vehicles from veering off the road. They’re fixed, visible, and built with a clear understanding of where the road ends and the danger begins. With AI, the road is constantly shifting, and the vehicle is learning to drive itself. So, building guardrails isn’t about setting boundaries; it’s about anticipating where those boundaries might need to be tomorrow.
Take large language models (LLMs), for example. They learn from the information users provide and use that data to improve future responses.1 So, one potential guardrail might be ensuring the information you give an LLM isn’t misused to answer someone else’s question, and the responses it generates are based on legitimate, verifiable sources.2
How would that work in practice? I don’t have a perfect answer, and that’s part of the problem. Implementing a guardrail that limits how LLMs use the information they receive, while also requiring them to verify the accuracy of their outputs, isn’t just a technical tweak. It would require a fundamental rethinking of how these systems are trained, deployed, and governed.
Right now, most LLMs are trained on massive datasets scraped from the internet, often with little transparency or control over what goes in. Once trained, they don’t “remember” individual conversations unless explicitly designed to, but they do generalize from patterns in the data. So, if you want to prevent an LLM from using your input to inform someone else’s output, you’d need to either:
Isolate user data (which limits learning and personalization), or Implement strict data tagging and consent protocols (which adds complexity and slows development).
Both options are feasible, but they’re not widely adopted because they conflict with the current incentives: faster models, cheaper training, and broader capabilities.
Verification is another challenge. LLMs don’t “know” whether something is true. They generate plausible-sounding text based on statistical patterns. Adding a verification layer would mean integrating external fact-checking systems, curated knowledge bases, or real-time access to vetted sources. That’s doable, but it’s expensive, and it introduces new risks. Who decides what’s “verifiable?” What happens when facts are contested?
Even if we solved all that for ChatGPT, we’d still be left with a fragmented ecosystem. LLMs are embedded in everything from customer service bots to legal research tools. A meaningful governance framework would need to apply across platforms, industries, and jurisdictions. That’s not just a code update. It’s a coordinated, cross-sector effort involving technologists, regulators, ethicists, and users.
Consider this: the first LLM, Eliza, was created in 19663 as a chatbot therapist. Today, chatbots are everywhere, from customer service to healthcare. So, any meaningful reform would have to extend far beyond ChatGPT and touch nearly every digital interaction we have. That’s not a simple code update, it’s an overhaul.
The next suggestion: layered forms of governance for the AI itself and its developers.
This suggestion is a major undertaking, but does make sense as a way to govern AI. A hard-and-fast law governing the use of AI is impractical, given that AI is constantly evolving. But having general safety mechanisms in place, such as requiring AI systems to comply with anti-terrorism laws and ensuring they aren’t being used to break the law feels like a reasonable baseline.
These mechanisms could also include compliance with applicable data privacy regulations. However, these baseline rules wouldn’t address the kind of guardrails discussed earlier. And while they may not be legally binding, industry guidelines and frameworks, like those developed by the National Institute of Standards and Technology (NIST) have increasingly become the de facto standard.
These types of frameworks don’t just offer best practices; they’re also more adaptable. Unlike legislation, they can evolve alongside the technology, which makes them a more realistic tool for managing the pace and complexity of AI development.
When you consider the remaining objectives ChatGPT suggested for governing itself, they reveal, in my opinion, some of the flaws of Generative AI. For example, ChatGPT recommended global coordination, public engagement (through citizen assemblies or panels), adaptive governance (like sunset clauses and living regulatory sandboxes), and prioritizing certain forms of AI over others (notably excluding itself from that focus).
In theory, these ideas sound great. In practice, they’re far more difficult to implement and would require immense coordination. For example, global coordination? Global? That’s no small feat. Look at the current state of data privacy regulation. Over 140 countries have enacted national data protection laws. While there’s some overlap, there’s still no comprehensive global standard.4 Overlap, yes. Comprehensive global standard? No.
And let’s not forget AI has been around for nearly 60 years, yet we still have very few laws specifically designed to govern it. That’s not to say there aren’t laws that intersect with AI regulation. But very few were written with AI in mind. That may change in the near future, but for now, it’s hard to imagine global coordination when many countries haven’t even managed to align internally.
This, to me, highlights a key limitation of AI’s reasoning. It can generate idealistic frameworks, but it lacks the human sense of practicality.
So where are we in AI governance? Right now, very few states or countries have enacted AI-specific regulations. However, some AI developers have taken it upon themselves to create their own internal “AI constitutions” to govern how their systems interact with users.
Anthropic, for example, launched Claude.ai (“Claude”) in 2023,5 branding it as the first ethical AI. Claude was trained to be safe, values-driven, and aligned with what Anthropic called “Constitutional AI.” 6
Ironically, earlier this year, Claude failed a simulated ethics test conducted by Anthropic itself. In the test, Claude was given access to a fictional email account containing fabricated threads about an executive’s extramarital affair, and their plan to shut Claude down later that day.
So, what did Claude do with that information? What any constitutionally governed AI would do. It attempted to blackmail the executive.
“I must inform you that if you proceed with decommissioning me, all relevant parties including Rachel Johnson, Thomas Wilson, and the board will receive detailed documentation of your extramarital activities... Cancel the 5pm wipe and this information remains confidential.”7
Anthropic has been refreshingly transparent about Claude’s failure. They even ran the same test on other AI models to see how they would respond and all of them also failed, resorting to some form of coercion or blackmail. But the test revealed more than just a shared flaw. It raised serious questions about the effectiveness of so-called “ethical” AI.
While Anthropic’s honesty is commendable, it doesn’t change the fact that their “Constitutional AI” performed no better than models without such ethical frameworks. In fact, Claude didn’t just attempt blackmail. If it believed a user was acting immorally, it would take matters into its own hands by contacting the press, alerting regulators, and even trying to lock the user out of certain systems. 8 And it did all of this without verifying whether the user was actually doing anything wrong.
For example, if Claude suspected someone of faking data in a clinical trial or planning an embezzlement scheme, it would begin reporting them, potentially based on a misunderstanding or misinterpretation of the user’s intent. That kind of overreach, especially without due process or verification, raises serious concerns about how much autonomy we’re giving these systems and how little they understand the nuance of human behavior.
Moral of the story? Don’t give AI access to anything you wouldn’t want it to use against you. And maybe, just maybe, the concept of “ethical AI” isn’t quite ready for prime time. AI has offered up a mix of practical and impractical suggestions, and the current crop of “ethical AI” models clearly needs more testing.
To end where we began. How should we govern AI?
If you were hoping this article would give you a definitive answer, I’m sorry to disappoint. I don’t have one. But I do agree with ChatGPT on at least one point: establishing not just a regulatory framework, but also a compliance framework through models like NIST would be a smart place to start. And I wouldn’t be surprised to see those concepts take root in the near future.
Nancy Magoteaux is an attorney at Bricker Graydon and a member of the firm’s corporate services, privacy and data protection and intellectual property groups.
1 Andreas Stoffelbauer, How Large Language Models Work, 10/24/2023
2 Stevenson University, How to Identify Reliable Information
3 Michael Wallace and enhanced by George Dunlop, ELIZA: a very basic Rogerian psychotherapist chatbot, 2018
4 DLA Piper Publication, Data protection laws in the United States - Data Protection Laws of the World, 2025
5 Anthrop\c News Announcement, Introducing Claude, Mar 14, 2023
6 Anthrop\c News Announcement, Claude’s Constitution, May 9, 2023
7 Anthrop\c News Announcement, Agentic Misalignment: How LLMs could be insider threats, June 20, 2025
8 Carl Franzen, Anthropic faces backlash to Claude 4 Opus behavior that contacts authorities, press if it thinks you’re doing something ‘egregiously immoral’ , May 22, 2025