NeurIPS 2026 Physics-grounded image-to-video generation

Seeing the Unseen Unified Visible–Invisible Motion for Physically Consistent Video Generation

A learned particle simulator predicts how the scene in a single image will move. A visible–invisible motion field hands that motion to a video diffusion model — including the surfaces that only come into view after objects move.

Paper Code Model BibTeX

Mingyang Bao1 Fangda Ye1 Xihao Chen1 Bobo Li1 Jason Li2 Shengqiong Wu3,✉

1National University of Singapore 2Nanyang Technological University 3University of Oxford

✉ Corresponding author

Teaser: two examples, a wooden ball colliding with a candle and a three-block domino chain. For each, rows show the generated frames, the reference-visible motion video (rainbow tracking points on regions seen in the input frame) and the future-emergent motion video (tracks on surfaces that only appear later, such as the candle's base as it tips over).
Seeing the Unseen. Our framework predicts physically consistent motion from a single image by modeling both reference-visible and future-emergent trajectories. By reasoning about occluded dynamics and structured interactions, it enables accurate simulation of complex events such as collisions and domino chains.
Learned physics

95.92%

PGDS learns physical dynamics from data

Average strict Acc-V of the particle simulator over seven Physion++ test scenarios — with 0 manually specified physical parameters.

Visible + emergent motion

2 motion videos

Motion you can see, and motion that is about to appear

VIMF splits guidance into a reference-visible and a future-emergent tracking video, injected through zero-initialized latent channels (48 → 144).

Seven scenarios

4.160 / 4.394

Highest Semantic Adherence on all seven scenarios

Collision SA / PC, versus 2.917 / 3.067 for the Wan2.2-TI2V-5B backbone. Best PC on six of the seven, against open-source, physics-aware and closed-source baselines.

Human study

+44.0%

Preferred by human annotators

Overall-preference margin over Wan2.2-5B in pairwise evaluation, alongside +42.0% in instruction-following.

The problem

Video models look real.
Their physics often does not.

Recent video generators reach impressive visual quality, yet they often fail to produce physically consistent dynamics, because the underlying physical processes are never modeled explicitly. Existing approaches either rely on simulators with strong assumptions or use coarse semantic guidance — and both struggle to inject physical priors into the generator.

Seeing the Unseen splits the problem in two: learn the physical dynamics, then inject them into video synthesis. A lightweight Particle-based Graph Dynamics Simulator (PGDS) predicts plausible 3D trajectories, and a Visible–Invisible Motion Field (VIMF) turns them into complete, structured motion guidance.

  1. i

    Projection discards depth

    Turning 3D trajectories into 2D flow or tracks loses depth and spatial structure.

  2. ii

    Only the reference view

    2D conditions cover regions visible in the input frame, not content that is occluded or unseen at first.

  3. iii

    Partial approximations

    Optical flow, tracking videos and sparse keypoints miss object-level and structure-level dynamics.

Method

Lift the image, simulate the physics, guide the video.

Given an input image, we lift it into a simulatable 3D particle scene. PGDS models object interactions and predicts physically plausible 3D trajectories, which are rendered into reference-visible and future-emergent motion that guides a fine-tuned video diffusion model.

Pipeline overview: the reference image is split into main objects and background and lifted into a 3D particle representation; a vision-language model estimates velocities from the text prompt; the Particle-based Graph Dynamics Simulator rolls out a 3D motion trajectory, which is rendered into reference-visible and future-emergent motion videos that, with the reference image and text prompt, condition the video generation backbone.
Figure Overview of the framework. The input image is lifted into a 3D particle representation; a particle-based graph dynamics simulator predicts physically plausible 3D motion trajectories, which are rendered into reference-visible and future-emergent motions to guide physically consistent video generation.
  1. 1

    3D lifting

    MoGe-2 predicts a metric point map. SAM3 segments the foreground objects from text prompts, and SAM 3D Objects reconstructs each one with its metric pose. The ground plane is fitted with RANSAC and the merged objects are converted into simulation particles; a vision-language model estimates initial velocities from the prompt.

  2. 2

    PGDS rollout

    Particles become graph nodes, connected to neighbors within radius r. Node features encode material, object-relative offset and velocity state; after T rounds of message passing a decoder predicts each particle's velocity, which updates positions and rolls out the trajectory. PGDS is trained on Physion++ with a velocity MSE loss.

  3. 3

    VIMF-guided generation

    The rollout is projected into a reference-visible and a future-emergent tracking video. Both are encoded by the frozen 3D VAE of Wan2.2-TI2V-5B and concatenated with the noisy latent; the first-frame latent stays clean for appearance conditioning.

~3000
particle nodes per scene
r = 0.08
neighbor radius for graph edges
10
rounds of message passing
48 → 144
input latent channels, tracking weights zero-initialized
Architecture of the physics-guided video generation model: the visible and future-visible motion fields are encoded by a frozen 3D VAE and concatenated with the noisy video latent along the channel dimension before the diffusion transformer.
Figure Architecture of the physics-guided video generation model. The visible and future visible motion fields are encoded by a frozen 3D VAE and concatenated with the noisy video latent along the channel dimension. A zero-initialized injection strategy preserves the pretrained model behavior while progressively incorporating motion guidance.
PGDS architecture Encode – Process – Decode
PGDS architecture: particle states are turned into a proximity graph, node and edge encoders embed material and geometric relations, a message-passing processor propagates interaction effects, and a decoder predicts particle velocities used to roll out future dynamics.
Architecture of the Particle-based Graph Dynamics Simulator. A graph is constructed from spatial proximity; node and edge features capture material properties and geometric relations; a message-passing processor iteratively propagates interaction effects; a decoder predicts particle velocities, which update positions and roll out future dynamics.

Visible–Invisible Motion Field

Guide what is visible and what is about to appear.

Optical flow or sparse point tracks only describe regions observable from the reference view. VIMF decomposes physical motion into two complementary tracking signals, so the generator also receives motion for occluded surfaces and objects that become visible in later frames.

Stream 1

Reference-visible motion Mvis

Records the motion of points that are visible in the input frame.

Reference-visible tracking example
Training
4,900 query points on a regular grid over the first frame are tracked through the video with SpaTrack2; only visible points inside the image are drawn.
Inference
Visible mesh vertices (backface culling + viewport clipping) are matched to an image grid with a KD-tree and moved by per-frame rigid transforms solved via SVD from the PGDS rollout.
Stream 2

Future-emergent motion Mfv

Captures points that are occluded or outside the reference view at first, but become visible in future frames.

Future-emergent tracking example
Training
8,100 candidate points per later frame are back-projected onto the first frame; those that fall outside it or behind its depth are kept as future-visible and tracked.
Inference
For vertices hidden in the first frame, the first frame where each becomes visible is recorded; grid downsampling keeps the earliest point per cell, and positions are locked before it appears.

Both tracking videos are encoded by the same frozen 3D VAE and concatenated with the noisy video latent along the channel axis. Patch-embedding weights for the original 48 channels are copied from the pretrained checkpoint, while those for the two tracking streams start at zero — so fine-tuning begins from the unmodified model and adds motion guidance gradually.

Ablation · Video comparison

Ablation on future-emergent motion signals.

Full model (Ours)Reference-visible + future-emergent motion
Visible-only variantReference-visible motion only

Our full model uses both reference-visible and future-emergent tracking signals. The visible-only variant uses reference-visible tracking alone.

With visible tracking alone, the model often produces ghosting or missing content: visible trajectories describe only pixels observed in the reference frame, and say nothing about regions exposed after objects move. Future-emergent tracks supply that missing motion, so newly appearing content is reconstructed coherently.

Qualitative results

Contacts that propagate. Objects that stay solid.

Baselines frequently show object interpenetration, implausible trajectories, or never trigger the expected physical event. Our method produces correct collision responses and complete domino chains.

Comparison with baselines

Compare methods using the same reference image and prompt. Choose a scenario and play the videos side by side.

Domino · Video comparison

One scene. Six methods.

PromptThe right wooden block tips slowly to the left and bumps the middle block. The middle block rocks and falls into the left block, which then topples over to the left, completing a gentle domino chain on the wooden tabletop.

OursOur method
Wan2.2 14B
HunyuanVideo 1.5
CogVideoX 1.5
Wan2.2 5B
Veo 3.1

Same reference image and prompt. Read left to right, then top to bottom. Videos play at their original speed and duration.

Same trajectories, different injection

We convert the same PGDS-predicted trajectories into the representation each method expects and keep its original pretrained weights: optical flow for Go with Flow, 10 sparse tracking points for Wan-Move, and visible tracking videos for Diffusion as Shader.

  • Optical flow struggles to represent object motion under occlusions and complex interactions, causing motion errors and artifacts.
  • Visible-only trajectories lack motion for newly revealed regions when objects rotate or become occluded.
  • Wan-Move (Wan2.2-I2V-A14B backbone) does well on Translation but gives less accurate control in complex interactions.
See the injection ablation table ↓
Comparison with different physical prior injection methods: frames generated from the same PGDS trajectories by Go with Flow, Wan-Move, Diffusion as Shader and our model.
Comparison with different physical prior injection methods.

Benchmark

Seven physical scenarios. 350 samples.

Four primitive motion types — free fall, domino, collision and translation — and three compositions, each with 50 reference images and prompts, including multi-object scenes. Semantic Adherence (SA) measures alignment with the prompt in content and motion (e.g., force direction and location); Physical Commonsense (PC) measures whether the motion follows plausible dynamics. Both are scored 1–5 by VideoPhy2-AutoEval-7B; higher is better.

Scores by scenario

Domino · SA and PC for every method (1–5)

SA PC

Quantitative results on seven physical video generation scenarios

A highlighted value is the best in its column; “–” indicates unsupported cases. Click a scenario to show it in the chart above.

Physical prior injection strategies & visible-only ablation

All methods receive the same PGDS trajectories. Our visible-only and full variants share the same diffusion backbone; the full variant adds the future-emergent tracking video. Highlighted = best in column.

The visible-only variant scores lower than the full model in all four scenarios on both metrics, with the largest gains from dual-stream guidance on Domino and Collision.

VideoPhy2 subset

External evaluation on VideoPhy2 samples compatible with our setting. Highlighted = best.

Future-emergent guidance lifts both scores over the visible-only variant.

PGDS motion prediction on Physion++

Vel. RMSE over complete rollouts; Acc-V: share of particles within strict (0.05 m/s or 5%) or relaxed (0.10 m/s or 10%) velocity-error thresholds; energy bias = (Epred − EGT) / EGT × 100%.

Configuration efficiency of trajectory generation methods

Setup cost and generalization, with time and parameter estimates from public implementations.

PGDS needs no manual parameter tuning, keeps a competitive runtime and generalizes strongly to unseen shapes.

Pairwise human evaluation

Annotators compare our videos with each baseline on (a) instruction-following, (b) video quality — visual fidelity, temporal coherence, artifact severity — and (c) overall preference. Win / tie / lose rates in %.

+42.0%instruction-following over Wan2.2-5B
+38.0%video quality
up to +44.0%overall preference
Pairwise human evaluation bar charts for instruction-following, video quality and overall preference. Against Wan2.2 14B, Wan2.2 5B, CogVideoX1.5 5B, HunyuanVideo 1.5 and Veo 3.1, our method has a positive win-minus-lose margin in every comparison.
Pairwise human evaluation comparing our method against baseline models; win / tie / lose rates (%).

Citation

BibTeX

bao2026seeing.bib 
@inproceedings{bao2026seeing,
  title     = {Seeing the Unseen: Unified Visible--Invisible Motion for Physically Consistent Video Generation},
  author    = {Bao, Mingyang and Ye, Fangda and Chen, Xihao and Li, Bobo and Li, Jason and Wu, Shengqiong},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

Acknowledgements

This codebase builds on SAM-3D-Objects, SAM3, Physion and Physion-particles, SpaTrackerV2, Diffusion as Shader, VideoX-Fun, Wan2.2 and MiraData. We thank the authors for making their work publicly available.