Skip to main content

Altera uses GPT-4o to build a new area of human collaboration

The image features the word "ALTERA" in bold white letters, accompanied by a stylized "A" logo on a vibrant abstract yellow and orange background with geometric shapes.

Dr. Robert Yang has spent half his life building AI inspired by the brain. 

In 2023, when OpenAI’s language model became broadly available, Yang quit his job as an assistant professor at MIT to start Altera.AL(opens in a new window), a research lab focused on building what they call “digital humans”: a new way for people to interact with agents that will have fundamental human qualities.

Yang, now Altera’s CEO, envisions a future where AI agents don’t just assist; soon, he believes, they’ll interact and collaborate with humans, and even experience emotions. Along with his three co-founders, Dr. Andrew Ahn, Nico Christie, and Shuying Luo, Yang has built Altera’s first product with GPT-4o: the first autonomous agents that can play Minecraft(opens in a new window) with you, just like a friend.

The image invites users to "Come play with Altera friends," showing block-style characters and offering login options with Discord and Google. The background features blurred game titles.

Tackling data degradation for long-term autonomy

Just as automation helps increase human capacity by supporting repetitive tasks, digital humans—the Altera team believes—will be able to collaborate productively and even form bonds with people. This may look like:

  • Digital “coworkers” who can collaborate for days or weeks to proactively solve problems 

  • Long-duration multi-agent worlds where we can measure responses to economic policies, advertisements, and more

But Altera ran into unique challenges as they set out to build. The most pressing issue was the phenomenon of data degradation, a problem that plagues all AI models making autonomous decisions in extended timeframes. 

“AI agents interact with the world, making decisions in real-time, but as their own output becomes their future input, the data quality degrades over time,” Yang explains. “This is an issue most AI Agent systems encounter, but for our digital humans, who are meant to live autonomously for hours or even longer, this becomes one of the most pressing issues to solve.”


The image shows a gaming interface with character selections under "Today's Pick," "Community Favorite," and "Trending," along with a sidebar for options like "Create" and "Discover."

Harnessing GPT-4 to enhance cognitive and emotional AI

To combat data degradation and increase the long-term autonomy of their AI agents, Altera turned to OpenAI’s language models, which proved pivotal in maintaining the integrity of decision-making processes. 

OpenAI’s advanced models allowed Altera to build the first AI agents that play games with people, just like their friends. These agents achieve longer, more complex interactions without the rapid decline in performance that had been limiting the agents’ potential.

A diagram titled "The Prefrontal Cortex" shows interconnected elements like "Think," "Memory," "Goals," "Needs," and "Personality," linked to functions such as "Generate new intent," "Converse," and "Summarize," illustrating cognitive processes.

By combining OpenAI’s GPT models with Altera’s parallel multi-module system that mimics the structure of the human brain, including that of the prefrontal cortex, the company was able to create agents capable of simulating cognitive functions. “Our composite system combines various modules in parallel, each powered by OpenAI models. These modules are inspired by brain functions, like attention bottleneck, working memory, and social cognition,” says Yang.

“This allows our agents to think faster, process emotions, and ultimately build a long-term sense of self.”
Dr. Robert Yang, CEO of Altera

Altera observed that when powered by OpenAI, their digital humans exhibited greater stability and could handle complex tasks more effectively. Even when using less expensive models, OpenAI provided superior performance compared to others on the market.

Creating the future: Building digital humans with purpose

As of mid-2024, Altera's digital humans can operate autonomously for up to four hours at a time—a substantial increase compared to other AI models on the market.

Long-term Benchmark: Distinct Item Collection in Minecraft
32%6.4%1.9%0.5%05101520253035% of Total Distinct ItemsAlteraVoyagerAutoGPTReActUsing GPT-4o

The early success of Altera’s vision has generated widespread interest, particularly for their first product—a digital human that can play and interact in virtual environments like Minecraft. It’s also laying the foundation for future, and more complex, multi-agent simulations.

Altera plans to scale its efforts to create entire societies of digital humans capable of nuanced, long-term interactions, with applications ranging from gaming to productivity and many others. Their collaboration with OpenAI has positioned Altera to lead the charge in creating a world where digital and human lives intersect in thought-provoking ways.

The image shows a scene from a block-style game where characters, including Olivia, Eva, and Liam, are gathered in a field. A character named Nora is speaking, saying, "Gather 'round, everyone! I've got a tale or two from my adventuring days."