HN 日本語サマリー

← 一覧へ戻る
科学・技術

Lift4D:ワイルドな環境での4D再構築に向けた単一視点3D推定の調和

Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild (lift4d.github.io)

29 pointsby ilreb1 コメント

要約

Lift4Dは、単一のモノキュラー動画から、カメラで観察されていない領域を含む動的オブジェクトの完全な形状、外観、変形を再構築するテスト時最適化フレームワークです。既存の単一視点3D再構築モデルを適応させ、一時的に一貫性のあるフレームごとの予測を生成し、これを変形可能な3D Gaussian Splatting表現の初期化に利用します。このアプローチにより、大規模な変形やオクルージョンを伴う複雑なワイルドなシナリオでも、従来の4D再構築手法を大幅に改善します。

全文翻訳

Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild

Carnegie Mellon University

*Equal co-advising.

Lift4D reconstructs the full geometry, appearance, and deformation of dynamic objects in a scene, including regions never observed by the camera, from a single monocular in-the-wild video.

Abstract

Reconstructing complete dynamic objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict per-frame 3D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well.

We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then “sculpt” this representation to match the input video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.

Reconstructing Complete 4D In-the-Wild

Pick a scene below to explore its complete 4D reconstruction in the interactive viewer. Click and drag to orbit; scroll to zoom.

Click a thumbnail to switch the scene. Please be patient as some scenes are large.

Methodology

Single-view Reconstruction Prior

Image-to-3D DiT

[画像: Image-to-3D DiT][画像: Image-to-3D DiT]

Image-to-3D DiT

[画像: Image-to-3D DiT][画像: Image-to-3D DiT]

Image-to-3D DiT

[画像: Image-to-3D DiT][画像: Image-to-3D DiT]

Causal Latent Propagation

↓

Gaussian SplatDecoder

↓

[動画: Gaussian SplatDecoder の出力]

Per-frame 3D Reconstruction

[動画: フレームごとの3D再構築]

Input video

[動画: 入力ビデオ]

Complete 4D Reconstruction

[動画: 完全な4D再構築]

Canonical Gaussians

[動画: 標準ガウシアン]

[動画: 変形ノード]

Deformation Nodes

Fine App. Deformation Nodes

Occlusion-aware Appearance Loss

[画像: オクルージョン認識アピアランス損失][画像: オクルージョン認識アピアランス損失][画像: オクルージョン認識アピアランス損失]

Inpainting

←

[画像: インペインティング][画像: インペインティング][画像: インペインティング]

Scene Depth / Per-frame 3D

[画像: シーン深度 / フレームごとの3D][画像: シーン深度 / フレームごとの3D][画像: シーン深度 / フレームごとの3D]

Occlusion-inpainted Frames

[画像: オクルージョン補完済みフレーム][画像: オクルージョン補完済みフレーム][画像: オクルージョン補完済みフレーム][画像: オクルージョン補完済みフレーム]

Noisy Deformed 3DGS Renders

↓

Novel-view Diffusion Prior

↓

[画像: 新規ビューの拡散事前知識][画像: 新規ビューの拡散事前知識][画像: 新規ビューの拡散事前知識][画像: 新規ビューの拡散事前知識]

Novel View Samples

([画像: 新規ビューサンプル]−[画像: 新規ビューサンプル])

Rendering Supervision

[画像: L_rec] [画像: L_app]

From a monocular input video, an image-to-3D DiT produces a temporally consistent per-frame 3D reconstruction through causal latent propagation, where each frame’s 3D latent is initialized by mixing fresh noise with the previous denoised latent, and the outputs are decoded into independent sets of Gaussian splats. We consolidate these per-frame predicted sets into a single 4D complete Gaussian Splat reconstruction, represented by canonical Gaussians animated by two sets of sparse deformation nodes. The first set is fit to the per-frame outputs through a reconstruction loss (ℒrec) on the per-frame reconstructed geometry, and the appearance is then refined by optimizing the color as well as a second set of fine appearance deformation nodes against occlusion-inpainted frames and rendering loss: the 4D reconstruction is rendered from random novel views and noised, and a novel-view diffusion prior denoises them, conditioned on the per-frame frames that have their occlusions inpainted using the per-frame 3D outputs. The resulting denoised novel-view sample distillation together with a rendering loss on the visible pixels supply an appearance supervision signal (ℒapp) that aggregates visible details across frames and hallucinates in occluded and unobserved regions.

Comparisons

Lift4D outperforms prior 4D reconstruction baselines on both synthetic and in-the-wild footage, delivering complete temporally coherent geometry, sharper appearance, and more accurate motion even under heavy occlusion.

BibTeX

@article{litman2026lift4d,
  author  = {Litman, Yehonathan and Ma, Xiaoxuan and Shah, Manan and Ugrinovic, Nicol{\'a}s and Kitani, Kris and De la Torre, Fernando and Tulsiani, Shubham},
  title   = {Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild},
  journal = {arXiv preprint arXiv:2606.23688},
  year    = {2026},
}