Over the past few years, many of us have gotten a crash course in what we now call artificial intelligence—but really, it has mostly been a crash course in large language models. Increasingly, however, LLMs are no longer the only category of AI drawing high expectations, massive funding rounds, and significant research and product development.
Over the past year, we’ve seen a plethora of new announcements in a category labeled “world models,” and you’ll likely see more movement there in the coming months and years.
Instead of or in addition to working with language, world models aim to lay the groundwork for AI systems that are capable of simulating the physical world, or at least a useful approximation of it.
To examine what’s different and important about this idea, Ars spoke with three expert practitioners working on world models and related technologies: Vincent Sitzmann from MIT, Anastasis Germanidis from Runway, and Ben Mildenhall from World Labs.
From these conversations, we learned that while LLMs-as-a-product started with an interface (chat) and then sought a use case, the big players in world models right now are arguably working in the other direction: They’re starting with specific use cases and applications in robotics, research, and asset generation, but it’s unclear exactly how the interfaces, systems, and tools will ultimately look.
As you’ll soon see, there are many parallels between LLMs and world models in terms of architecture and how people expect them to improve over time. For some, though, they’re seen as a potential answer to the limitations of LLMs, even though work on them predates that contemporary narrative.
“The idea that you’re going to extend the capabilities of LLMs to the point that they’re going to have human-level intelligence is complete nonsense,” former Meta chief AI scientist Yann LeCun told Wired earlier this year. LeCun has made waves with an opinion that some working in AI and LLMs see as contrarian, but he’s actually speaking for a sizable segment of the field.
See also Fei-Fei Li, the computer vision pioneer who co-founded World Labs, one of the new companies working on world models. In a Substack post late last year, she wrote:
Today, leading AI technology such as large language models (LLMs) have begun to transform how we access and work with abstract knowledge. Yet they remain wordsmiths in the dark; eloquent but inexperienced, knowledgeable but ungrounded. Spatial intelligence will transform how we create and interact with real and virtual worlds—revolutionizing storytelling, creativity, robotics, scientific discovery, and beyond. This is AI’s next frontier.
LeCun and Li’s ventures are built on these ideas, so it’s not surprising they’d say these things. But you’ll also see similar sentiments from some prominent figures still working primarily with LLMs.
“I think we’re in an LLM bubble, and I think the LLM bubble might be bursting next year,” said Clem Delangue, the CEO of Hugging Face—a platform that hosts repositories of LLMs of all stripes.
“But ‘LLM’ is just a subset of AI when it comes to applying AI to biology, chemistry, image, audio, [and] video,” Delangue added, speaking at a conference. “I think we’re at the beginning of it, and we’ll see much more in the next few years.”
Over just the past few months, world models have advanced from a research topic (which they still are, of course) to the basis for new commercial projects and huge funding rounds. A few key examples:
These and similar efforts have received substantial funding. World Labs and AMI reportedly raised around $1 billion each in February and March, respectively, and Runway also raised $315 million in February.
Some of the activity around world models is at least in part aimed at ostensibly building the foundations of AGI or superintelligence, but most people working on them are talking about practical applications: training, testing, and driving robots; generating 3D assets for game development and film production; scientific simulation and modeling; and so on.
It’s important to note that “world models” is an umbrella term that is often thrown around without a clear definition, though.
“It’s definitely an overloaded term,” Vincent Sitzmann told me in a lengthy conversation about the research and concepts underlying world models.
Sitzmann is an assistant professor at MIT who has published research on neural rendering, visual computing, and robotics. He leads the Scene Representation Group within MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).
When asked to give a definition, he simply described a world model as any model that takes in an interaction, “and given that interaction, it enables you to simulate what would happen next in some environment.”
When announcing its GWM-1 family of models in December, Runway defined a world model as “an AI system that builds an internal representation of an environment and uses it to simulate future events within that environment.” Further, “the aim of general world models is to represent and simulate a wide range of situations and interactions—like those encountered in the real world,” Runway added.
I also spoke with Ben Mildenhall, co-founder of World Labs, a former Google computer vision and physics researcher and co-creator of neural radiance fields (NeRF)—a method for constructing navigable 3D scenes from 2D images in a format that is differentiable and therefore useful in machine learning contexts.
“The key things that would distinguish it from an LLM are demonstrating degrees of spatial and—maybe for lack of a better word—continuous understanding,” he said. “A very distinguishing aspect of interacting with an LLM is they are turn-based.” Meaning, users type some text, there’s a pause, and then they get a block of text back. By contrast, he sees a world model as a synchronous, real-time system.
“Something that would define a world model is the degree of freedom that you have in interacting with the spatial world, where you do not have this mediated linear journey of A then B then A then B then A then B,” he said. When a user or agent is utilizing a world model, they are “actually able to interact with it like it is some sort of world and you are taking continuous actions” where “there are parallel things happening at the same time.”
Mildenhall’s co-founder Fei-Fei Li has written that she believes there are three criteria that define a world model: World models “can generate worlds with perceptual, geometrical, and physical consistency,” “are multimodal by design,” and “can output the next states based on input actions.”
The phrase itself is not new; it has long appeared in reinforcement learning and robotics to describe models that predict environment dynamics. What is new is the attempt to scale that idea into general-purpose, generative systems trained on massive visual and multimodal data. The huge funding rounds are relatively new, too.
The truth is that there are numerous approaches and definitions. “Ask Runway, ask us, ask whoever—we’re all gonna give you a little bit of a different term,” Mildenhall said.
Further, it’s important to be aware that “world model” is becoming a branding term as much as a technical one. It’s used as a marketing label that can cover anything from action-conditioned video generation to 3D asset creation to robot policy evaluation. While there is conviction that one foundation model ought to be able to solve for a lot of these things, companies are using the term while solving different technical problems with varying approaches.
“Today, what most people mean when they say ‘world model’ is generating pixels, so, like, generating a realistic video conditional of the actions,” Sitzmann said.
