Altera uses GPT-4o to build a new area of human collaboration

Dr. Robert Yang has spent half his life building AI inspired by the brain. 

In 2023, when OpenAI’s language model became broadly available, Yang quit his job as an assistant professor at MIT to start Altera.AL(opens in a new window), a research lab focused on building what they call “digital humans”: a new way for people to interact with agents that will have fundamental human qualities.

Yang, now Altera’s CEO, envisions a future where AI agents don’t just assist; soon, he believes, they’ll interact and collaborate with humans, and even experience emotions. Along with his three co-founders, Dr. Andrew Ahn, Nico Christie, and Shuying Luo, Yang has built Altera’s first product with GPT-4o: the first autonomous agents that can play Minecraft(opens in a new window) with you, just like a friend.

Just as automation helps increase human capacity by supporting repetitive tasks, digital humans—the Altera team believes—will be able to collaborate productively and even form bonds with people. This may look like:

• Digital “coworkers” who can collaborate for days or weeks to proactively solve problems 

• Long-duration multi-agent worlds where we can measure responses to economic policies, advertisements, and more

But Altera ran into unique challenges as they set out to build. The most pressing issue was the phenomenon of data degradation, a problem that plagues all AI models making autonomous decisions in extended timeframes. 

“AI agents interact with the world, making decisions in real-time, but as their own output becomes their future input, the data quality degrades over time,” Yang explains. “This is an issue most AI Agent systems encounter, but for our digital humans, who are meant to live autonomously for hours or even longer, this becomes one of the most pressing issues to solve.”

To combat data degradation and increase the long-term autonomy of their AI agents, Altera turned to OpenAI’s language models, which proved pivotal in maintaining the integrity of decision-making processes. 

OpenAI’s advanced models allowed Altera to build the first AI agents that play games with people, just like their friends. These agents achieve longer, more complex interactions without the rapid decline in performance that had been limiting the agents’ potential.

By combining OpenAI’s GPT models with Altera’s parallel multi-module system that mimics the structure of the human brain, including that of the prefrontal cortex, the company was able to create agents capable of simulating cognitive functions. “Our composite system combines various modules in parallel, each powered by OpenAI models. These modules are inspired by brain functions, like attention bottleneck, working memory, and social cognition,” says Yang.

Altera observed that when powered by OpenAI, their digital humans exhibited greater stability and could handle complex tasks more effectively. Even when using less expensive models, OpenAI provided superior performance compared to others on the market.

As of mid-2024, Altera's digital humans can operate autonomously for up to four hours at a time—a substantial increase compared to other AI models on the market.

The early success of Altera’s vision has generated widespread interest, particularly for their first product—a digital human that can play and interact in virtual environments like Minecraft. It’s also laying the foundation for future, and more complex, multi-agent simulations.

Altera plans to scale its efforts to create entire societies of digital humans capable of nuanced, long-term interactions, with applications ranging from gaming to productivity and many others. Their collaboration with OpenAI has positioned Altera to lead the charge in creating a world where digital and human lives intersect in thought-provoking ways.