Video ↔ World A survey

Reading, Writing, Sharing, and Interacting with the World through Video

A Survey Toward Unified Video World Models

Shengqiong Wu1,* Meng Luo2,* Bobo Li2 Ziwei Liu3 Shuicheng Yan2 Ming-Hsuan Yang4 Philip Torr1 Hao Fei1,†

1 University of Oxford2 National University of Singapore3 Nanyang Technological University4 University of California, Merced

* Equal contribution   † Project lead & correspondence: Hao Fei

Survey overview ↓GitHub ↗669 references
Reading infers world state from video, writing generates video from conditions, sharing couples reading and writing, and interacting evolves state under actions with repeated video feedback.
Four operations between video and world state: reading, writing, sharing, and interacting.

The world-modeling lens

A video is a trace of a world.

Video foundation models understand scenes, generate futures, couple perception with creation, and respond to actions. This survey connects these capabilities through the relationship between observed video and an evolving world.

One world. Partial views.

Video ↔ World
The paper's illustrated landscape, with cameras observing a moving object from partial views.
A continuous world gives rise to a partial video trace. Views crop the same illustrated scene; the purple object can move out of view or behind a building. Conceptual animation, not model output.
V

Observed video

A partial, lossy, temporally ordered record of the world.

W

Latent temporal world

Entities, relations, and physical properties evolving over time.

st

Represented state

What a model infers and maintains. It can be incomplete or wrong.

The field at a glance · 2017–2026

Four paths, a shared world.

Selected milestones in video understanding, generation, shared representations and workflows, and world prediction and control.

The original paper timeline, 2017–2026.
Open timeline ↗

Four operations on the video–world relationship

§4 · Video comprehension

Reading the World

Infer semantic, spatial, temporal, physical, and causal properties
from a partial video trace.

Contrastive, masked, predictive, and generative learning construct different supervision targets for video representations.
Video representation learning. Contrastive, masked, predictive, and generative learning construct different supervision targets for video representations.

From recognition to maintained state

Video encoders, Video-LLMs, test-time scaling, and video agents offer different ways to gather and retain evidence—from short clips to long, streaming observations.

What establishes understanding?

Does an answer actually depend on the video? Control for text-only, single-frame, shuffled-order, and restricted-context shortcuts.

Representation learning · Video-LLMs · Video agents · Applications

Reading literature 152 ↗

§5 · Video generation

Writing the World

Render a video trace from a partial specification:
text, images, audio, motion, or other conditions.

Generative models Latent representations Planning & agents Control mechanisms
Diffusion and flow matching operate over video latents; autoregressive modeling follows an ordered sequence, while masked prediction fills selected positions from context.
Generative models. Diffusion and flow matching operate over video latents; autoregressive modeling follows an ordered sequence, while masked prediction fills selected positions from context.
Continuous, discrete, hybrid, and semantically aligned designs shape compression, reconstruction, sequence modeling, and unification with understanding.
Latent representations. Continuous, discrete, hybrid, and semantically aligned designs shape compression, reconstruction, sequence modeling, and unification with understanding.
LLMs and MLLMs plan for visual generators; agentic systems connect scripting, generation, evaluation, and revision through a controller.
Planning & agents. LLMs and MLLMs plan for visual generators; agentic systems connect scripting, generation, evaluation, and revision through a controller.
Control enters through inputs, attention maps, latent states, block features, parameters and embeddings, or the sampling update.
Control mechanisms. Control enters through inputs, attention maps, latent states, block features, parameters and embeddings, or the sampling update.

From conditions to trajectories

A prompt fixes only part of a scene. A generator resolves the remaining appearance, motion, and events while preserving the identities and constraints the condition supplies.

More than plausible frames

Does the trace follow the condition, maintain a consistent world, and support its intended use? Visual quality alone does not establish these properties.

Representations · Generative models · Control · Editing & applications

Writing literature 128 ↗

§6 · Unified video models

Sharing the World

Couple understanding and generation through common
representations, parameters, interfaces, or objectives.

Unified architectures Learning paradigms
Discrete tokens, continuous latents, modular semantic bridges, and agentic task states provide four places for understanding and generation to meet.
Unified architectures. Discrete tokens, continuous latents, modular semantic bridges, and agentic task states provide four places for understanding and generation to meet.
Isolated, dual, co-learning, and synergistic learning differ in how they connect the two directions. These paradigms can coexist.
Learning paradigms. Isolated, dual, co-learning, and synergistic learning differ in how they connect the two directions. These paradigms can coexist.

Coverage, sharing, synergy

Supporting both tasks is capability coverage. Sharing machinery is an architectural claim. Demonstrating that the directions improve one another requires controlled evidence.

Isolate the benefit of coupling

Compare reading and writing with matched data, capacity, compute, and supervision. Test whether cross-direction information flow actually improves either task.

Unified architectures · Joint learning · Editing · Generative reasoning

Sharing literature 60 ↗

§7 · Video world models

Interacting with the World

Model how an established state evolves—and how actions,
controls, and new observations change that evolution.

Prediction space (observations or latent states) crosses transition interface (passive, action-conditioned, or interactive). The same model can occupy different cells in different training and use settings.
A 2 × 3 taxonomy of video world models. Prediction space (observations or latent states) crosses transition interface (passive, action-conditioned, or interactive). The same model can occupy different cells in different training and use settings.

What is predicted?

Observation-space models predict frames, video tokens, or video-decoding latents. Latent-state models predict video-derived features without required rendering.

When can we intervene?

Passive prediction has no action input. Action-conditioned prediction fixes controls beforehand. Interactive prediction accepts new controls after feedback.

“Interactive” describes the control interface, not the attention pattern. Video-decoding latents still belong to observation space.

Control · Memory · Responsiveness · Physical grounding · Planning

Interacting literature 119 ↗

Across the four operations

From plausible output to useful decisions.

Stronger world-modeling claims need stronger evidence.

  1. 01

    Plausible output

    Credible answers, frames, or features on the immediate task.

  2. 02

    Faithful conditions

    Dependence on observed evidence and supplied controls.

  3. 03

    Consistent state

    Persistent identity, layout, and physics; controlled transfer for sharing.

  4. 04

    Decision utility

    Improved planning, policy learning, or policy evaluation.

Datasets, benchmarks & evaluation →

§8 · Open challenges & future directions

What remains to be understood?

Seven challenges connect better methods with evidence of why they work.

01

Long-horizon world consistency

Memory must track how hidden state evolves, not only retrieve what was last visible.

A useful test

Vary rollout length and revisit distance separately; report degradation alongside memory, latency, and action-update rate.

Hidden state evolves during occlusion; replaying an old observation can miss that change.
Long-horizon world consistency. Hidden state evolves during occlusion; replaying an old observation can miss that change.
02

Physical and causal grounding

Changing an action or physical parameter should change the relevant outcome while leaving unrelated state intact.

A useful test

Compare paired interventions from the same initial state; locate the first violated constraint and when it appears.

Changing gravity alters the trajectory while unrelated scene properties remain fixed.
Physical and causal grounding. Changing gravity alters the trajectory while unrelated scene properties remain fixed.
03

Measurable synergy in unified models

Doing both tasks does not establish that understanding and generation improve one another.

A useful test

Compare coupled and uncoupled systems under matched data, capacity, compute, and supervision.

Two systems with matched resources differ only in whether information passes between understanding and generation.
Measurable synergy in unified models. Two systems with matched resources differ only in whether information passes between understanding and generation.
04

Standardized evaluation of world properties

Benchmarks use different conditions, horizons, and output spaces, so headline scores are difficult to compare.

A useful test

Align state properties and interventions across operations; report the protocol, budget, failures, and downstream effects.

Video and feature predictions need different probes, but both face the same decision test.
Standardized evaluation of world properties. Video and feature predictions need different probes, but both face the same decision test.
05

Scaling control from unlabeled video

Videos show motion but often omit the actions, units, and coordinate systems that caused it.

A useful test

Assess whether inferred controls predict interventions and transfer to a declared action interface, beyond passive reconstruction.

Action effects mix with camera and background motion.
Scaling control from unlabeled video. Action effects mix with camera and background motion.
06

Generalization across scenes and embodiments

Different cameras, bodies, and action spaces can turn an apparently reusable model into a setting-specific solution.

A useful test

Separate held-out scenes from held-out embodiments and evaluate their combinations under stated adaptation budgets.

A held-out scene–body pair tests whether familiar factors compose while preserving action effects.
Generalization across scenes and embodiments. A held-out scene–body pair tests whether familiar factors compose while preserving action effects.
07

Stochastic worlds and rare events

Rendering a rare event does not establish how likely it is. Diverse samples can still put probability mass in the wrong places.

A useful test

Use calibration, proper scoring rules, and valid-mode coverage at fixed sample budgets; report invalid outcomes separately.

Valid futures differ in frequency, and invalid and unreadable samples must be counted separately.
Stochastic worlds and rare events. Valid futures differ in frequency, and invalid and unreadable samples must be counted separately.

The reference collection

Explore the literature

669 works across the four operations, evaluation, and foundations.

Loading the reference library…

Updated October 23, 2026 · Full reading list on GitHub ↗

Survey figure

Open original ↗

Scroll to explore · Esc to close

BibTeX reference