TL;DR
Reconstruct geometry first, then predict each pixel's motion in its own camera frame.
- Object flow as the motion target. Static points are exactly zero, so there is no camera pose to infer.
- A geometry-first cascade. The motion decoder reuses DUSt3R's tokens and attends in 3D through 3D RoPE.
- A voxel Chamfer loss that gives unlabelled points a training signal.
Waymo
Kubric-F
Kubric-F
Compared with the best prior method.
Abstract
We introduce SM4RT, a multi-task cascade feed-forward model for dynamic scene reconstruction from RGB videos. While existing feed-forward approaches typically address geometry reconstruction and motion estimation separately -- and often predict motions in a single coordinate frame that must account for camera movement -- SM4RT solves these tasks sequentially by first reconstructing 3D geometry and then bootstrapping motion estimation from the predicted structure. By expressing each pixel's 3D motion in its corresponding camera frame, SM4RT reduces ambiguity arising from camera motion. To address the sparsity of real-world motion labels, SM4RT further incorporates a weakly supervised loss that regularizes unlabeled point motions using labeled point positions at the other time step. We evaluate motion estimation on three dynamic-scene datasets -- Kubric, Waymo, and Point Odyssey -- and SM4RT achieves the best performance on most metrics compared with prior baselines. We additionally benchmark geometry reconstruction on five dynamic scene datasets and show that SM4RT achieves competitive performance with state-of-the-art methods. Overall, our unified framework demonstrates that combining geometry reconstruction and motion estimation in a cascade not only avoids performance trade-offs but also enhances motion estimation through improved geometric reasoning.
Results
Explore the predictions in 4D
Each scene is a single SM4RT forward pass on two frames: the source frame's point map and its predicted object flow. Scrub through time, switch the direction, or colour points by how far they move.
Interactive 3D needs WebGL, which is not available in this browser. The point clouds can still be downloaded as .glb files.
Point maps and flow are up to scale, as in DUSt3R. Frame 1 → 2 moves frame 1's points to t₂ in camera 1's frame. Frame 2 → 1 moves frame 2's points back to t₁ in camera 2's frame, from a second pass with the inputs swapped.
Method
Geometry first, then motion
SM4RT is a multi-task cascade built on DUSt3R. It reconstructs dense 3D geometry from two frames, then estimates motion from that geometry, and it writes motion as object flow so camera motion never enters the target.
1Motion target: object flow
Let P₁(t, π) be the 3D positions of frame 1's pixels at time t, in camera frame π. Their motion from t₁ to t₂ can be written in three ways:
DUSt3R-style dynamic models (MonST3R, DPM, DynaDUSt3R) express both frames in camera 1, so frame 2's motion becomes cross-frame flow and the pose has to be inferred implicitly. SM4RT predicts object flow for each frame in its own camera. Static points are exactly zero, so a motion mask comes for free, and the static points' geometry gives the relative camera pose (PnP + RANSAC).
2Architecture: a two-stage cascade
Choose a component to see what it does. Swap the input order to get frame 2's motion from the same weights.
33D RoPE: attention follows 3D distance
To estimate motion, the decoder must know which pixels lie close together in 3D. Each token's position is the mean of its patch's predicted 3D points. Rotary embeddings of these positions bias attention towards tokens that are near in 3D, as in Lepard and 3D Diffuser Actor.
Hover over or tap a patch. With image positions, its neighbourhood bleeds into the floor behind the object. With predicted 3D positions, it stays on the object.
Illustration on SM4RT's predicted point map (Kubric-F): nearest neighbours by image distance (left) and by 3D distance (right).
4Training with sparse motion labels
Real-world motion labels are sparse. On Waymo, for example, they come from LiDAR points and 3D boxes. For an input pair, SM4RT minimises
ℒ = ℒconf(P1) + ℒconf(P2) + ℒconf(F1) + ℒSD + λ(t) ℒCD
DUSt3R's confidence-weighted regression, applied to labelled pixels only, for both point maps and the flow. All predictions share one scale factor g, so geometry and motion stay consistent.
Weakly supervised voxel Chamfer loss. It moves every pixel of frame 1 by its predicted flow and compares P₁ + F₁ with frame 2's ground-truth point map P̃₂(t₂, π₁), by nearest neighbours in both directions within each voxel. Pixels without motion labels get a signal too, and gradients reach both P₁ and F₁.
EMA self-distillation on static pixels. An exponential-moving-average teacher keeps the pretrained static geometry while the model learns motion.
Shrink-and-perturb on the flow head, which is initialised from the depth head. Fresh weights are mixed in at scheduled steps, so the head can leave the static prior.
Inside the voxel Chamfer loss
A toy scene seen from above, at t₂ in camera 1's frame. Rings are all of frame 1's pixels moved by their predicted flow, P₁ + F₁. Dots are frame 2's ground-truth points, P̃₂(t₂, π₁). Only three car pixels have motion labels (filled rings), so the rest of the car starts out smeared. No correspondences are needed. Inside each voxel, every ring is pulled to its nearest dot and every dot pulls its nearest ring. A voxel that holds only rings or only dots is skipped: that surface is seen in one frame only, because of occlusion or camera motion.
Drawn coarse for legibility. Training uses 1000 voxels per axis, and ℒCD is ramped in (λ from 0 to 4 on a sigmoid schedule) only once the point maps are accurate.
Data · static: WildRGBD, BlendedMVS · dynamic: Waymo, Kubric-F, Spring, Point Odyssey. Pairs are 1 to 15 frames apart; training runs 200 epochs.
Citation
@inproceedings{lo2026sm4rt,
title = {SM4RT: Cascaded Feed-forward Model for 4D Reconstruction},
author = {Lo, Pang-Chi and Lai, Jhao-Syun and Ke, Tsung-Wei},
booktitle = {British Machine Vision Conference (BMVC)},
year = {2026}
}Acknowledgements
Supported by the AMD University Program AI & HPC Cluster and NSTC Taiwan (115-2634-F-002-012, 115-2634-F-002-001). We thank Kuo-Guang Tsai for support on the AMD cluster. Funding to attend this conference was provided by the CMU GSA/Provost Conference Funding.