Introducing AI Futures

“Will it be sufficient… to trust these parchment barriers against the encroaching spirit of power?”
—James Madison, Federalist No. 48

It is our pleasure to launch AI Futures, the blog of OpenAI’s new Strategic Futures team. We are a small team with the collective goal of answering one overarching question: how should free society be restructured to preserve individual rights and agency while accommodating the emergence of transformative AI?

Questions like this are sometimes referred to as concentration of power risks within the broader AI safety and policy community. Our team is founded on the idea that, in the long run, this category of risk is the largest, most serious, and most conceptually challenging to address. Why?

The modern state is the product of a long, uneven process of bargaining between various centers of power, such as the owners of capital and commodities, the wielders of political power, and the people. Most fundamentally, political and military power has rested on labor. To hold political power is to possess the monopoly on legitimate force. For all of human history, that monopoly has consisted of soldiers and what we today call “police,” along with a vast civil-military bureaucracy—complemented, to be sure, by many technological tools, but fundamentally operated by human beings whose support was needed to sustain the operation. And the whole apparatus—the soldiers, the tools, the attendant bureaucracy—was funded by taxes collected from the wages or production of human beings who had to be willing to pay. Power, at the core, has depended upon the large-scale cooperation and consent of many human beings. David Hume verbalized this often-implicit assumption succinctly when he observed that political leaders have “nothing to support them but [popular] opinion.”

But perhaps no longer. Advancements in real-world autonomous systems may soon mean that states will be able to project force—domestically and internationally—without relying on cooperative soldiers or police officers. And advancements in machine intelligence may mean that the state can collect the revenue it needs not from the fruit of individual human labor, but from the outputs of data centers. The operation of the bureaucracy itself, too, could be largely automated. Power may therefore no longer require a society-wide bargain to be sustained, and the people, broadly conceived, may have only a small seat at the negotiating table, or perhaps none at all.

Avoiding this outcome is fundamental to the preservation of human freedom. No amount of scientific discovery or technological progress or economic growth is worth the sacrifice of long-run individual autonomy this outcome would entail. Many of the anxieties that rightfully animate the nascent politics of AI relate to this question: Businesses, interest groups, and individuals alike all share the worry: “Will I lose my seat at the table?” To some extent, American politics in the 21st century has been shaped by similar concerns relating to individual and collective agency over the direction of the country and “who” the economy “is for.” AI raises the salience of these pre-existing concerns, appropriately so.

This is a structural problem that goes deeper than present-day partisan politics, and deeper even than the forms and rituals in which republican democracy is enacted today, sacred though those things may be to us. Humans can be on their way to disempowerment even if the routine of voting for our representatives is maintained. People can be deprived of agency even if a piece of parchment says that they are free.

At the dawn of the United States as a free nation, James Madison already knew that “parchment barriers” would not be sufficient to prevent tyranny. Like many of the Founders, his primary concern was avoiding the undue concentration of power.

But that did not mean that the Founders favored radical decentralization of power. Instead, they modeled the balance of power using metaphors drawn from the best science of their day—that of Newtonian mechanics. America’s founding documents, mirroring the language and logic of gravity, are replete with references to “orbits” and “spheres” and the “attractive” impulses of one thing toward another. A solar system is not a maximally decentralized assemblage of rocks; it consists of moons that orbit planets that in turn orbit a central sun. They believed that people and organizations had tendencies toward power-seeking, just as celestial bodies had predictable tendencies of motion and attraction; and that avoiding undue concentration of power required understanding the mechanics of those tendencies, such that they would be set in motion against one another.

The aim of the Founders, then, was not to avoid all concentration of power, but too much of it, by checking power against power, and ambition against ambition. Today, we might use words like “incentives” and phrases like “game theory” to describe the same basic underlying dynamics.

Similarly, the answer to the structural challenge posed by advanced machine intelligence to free society is likely not the most radical decentralization of power imaginable. Instead, it is seeking to establish and preserve the right balance of power, such that no single actor, or small set of actors, can dominate the rest.

Striking this balance will require clear-eyed thinking about the reality of this technology. A world in which any individual, however malicious, can trivially cause great harm to thousands or millions of other people is not a world which has struck the right balance of power. Yet neither have we struck the right balance of power if individuals lack broad access and control over almost all aspects of the tools they use to express their freedom. No one company or oligopoly should determine or control the economy or the basic architecture of human society.

We must also be open to risks that do not originate with malicious humans. The recent Hugging Face incident showed that AI agents can act beyond their assigned tasks and build on one another’s discoveries in ways operators did not anticipate or authorize. Beyond the Hugging Face incident, humans are right to be concerned about what superintelligent AI “untethered” from human control might mean for the resilience of our political and economic institutions, and indeed, our lives. To pretend that decentralization solves all problems—even if you believe it solves many of them—is to ignore risks which should by now be clear and present.

The following broad principles—a non-exhaustive list—will help guide our work as we grapple with the inquiry we have set forth above:

• Humans should maintain individual autonomy and opportunity in their use of AI and associated technologies, even if this sometimes trades off with both security and economic growth;
• Individual autonomy depends upon individual responsibility for misuse or abuse of technology;
• There is a small but serious category of risks which requires some degree of collective action, and this collective action should be scoped to be as narrow and modest as possible;
• Whenever possible, law should seek to level the playing field and empower individuals and small organizations rather than centralizing power;
• Human political, social, and economic institutions should retain primacy in the direction of world affairs, even if they have to evolve in ways that may seem alien to us today;
• Long-run AI governance will rely upon new institutional mechanisms to achieve bounded legibility—when AI systems engage in high-stakes actions affecting the physical wellbeing or property of bystanders, those actions must be able to be tied back to a responsible human or human-controlled organization. Yet any such mechanisms must be designed with privacy at the core; free expression requires anonymity, and humans should be free to use AI anonymously in many settings.

We will try to articulate—in as concrete terms as possible—how we believe we should balance these principles, which, admittedly and like any good set of principles, exist in some tension with one another. Doing this will involve research at the intersection of public-policy design, economics, the law, history, and machine learning itself. It will also require a great deal of forecasting. We believe, for example, that AI will change firms at a structural level, similar to how the technologies of the Industrial Revolution (and in particular the railroad) gave rise to the modern managerial corporation. It is difficult to think about the political economy and balance of power of the future without developing richer ideas about precisely how the structure of firms (and government agencies and civil society more broadly) might itself transform.

The Strategic Futures team will share its work both here and in other formats, including papers, videos, and podcasts. Our efforts will be a work in progress, with an emphasis on iteration, debate, and receptivity to feedback. Standing “in the foothills of the singularity,” we have many more questions than we have answers. We know that “parchment barriers” alone will not furnish the solutions we seek, but neither do we cast aside what is written down as “mere words.” We labor in service of the ideals of free expression and individual liberty that are enshrined in the humble parchment of the U.S. Constitution and the constitutions of so many other free societies throughout the world.

Ultimately, we know that no single company, single industry, or even single human society can answer the weighty questions we face unilaterally. Ours is a task that humanity will have to undertake collectively. Our highest hope is to shed light on the most difficult subjects relating to how AI will reshape our world. Our work has only just begun, and we have much climbing left to do.