chenkangjie1123/VGGT-Diff

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis

_{Kangjie Chen^1*,  Xiangyu Li^1*,  Dongbin Zhang¹,  Chaoda Zheng¹,  Shijia Chen¹,  Jinhao Deng¹,  Hongbin Lin²,  Choo Sin Wai³  Minqi Wang²,  Minghao Yang³,  Dake Zhong³,  Guorui Song³,  Yu Zhang¹,  Xianming Liu¹,  Boyang Wang^1†}

_{¹ XPeng Motors    ² The Chinese University of Hong Kong    ³ Tsinghua University}

_{VGGT-Diff delivers strong novel-view synthesis across challenging in-domain and cross-domain viewpoints, outperforming existing generative NVS methods. Core model weights are now available on Hugging Face.}

• 🎉 Oct. 15, 2026: Stronger full-resolution NVS checkpoint released! Our new 480x832 scaling checkpoint, trained with 100K half-resolution steps followed by 6K full-resolution steps, is now available on Hugging Face. 🎉
• 🎉 Oct. 5, 2026: Model weights released! All three core checkpoints are now on Hugging Face. 🎉
• Sep. 29, 2026: arXiv paper and pose-free inference released.
• Sep. 27, 2026: Paper, project page, and code released.

The repositories contain the trained VGGT-Diff DiT and conditioning weights. Wan2.1, VGGT-Omega, and other external model components remain separate dependencies and are not redistributed with these checkpoints.

Download a checkpoint with the Hugging Face CLI, for example:

hf download chenkangjie1123/VGGT-Diff-Trajectory-480x832 \
  vggt-diff.safetensors config.json LICENSE README.md \
  --local-dir checkpoints/VGGT-Diff-Trajectory-480x832

Use --training-mode nvs with either NVS checkpoint. The trajectory checkpoint is the default model family for scripts/infer.py and scripts/infer_pose_free.py.

git clone https://github.com/chenkangjie1123/VGGT-Diff.git
cd VGGT-Diff

conda create -n vggtdiff python=3.10 -y
conda activate vggtdiff
pip install -e .
pip install -U huggingface_hub
pip install git+https://github.com/facebookresearch/vggt-omega.git@399d4d62935deb71cedb1e1c35b7a90413a6bee4

Request access to the VGGT-Omega checkpoint and download vggt_omega_1b_512.pt. Wan2.1 weights are downloaded automatically on first use, or can be supplied through --base-model-dir. A local model root should contain Wan-AI/Wan2.1-I2V-14B-480P/ and Wan-AI/Wan2.1-T2V-1.3B/google/umt5-xxl/; pointing to the Wan2.1-I2V-14B-480P directory itself is also supported when the tokenizer is available in the adjacent tokenizer/ tree.

The bundled example uses six source views, an 80-frame camera trajectory, and the released full-resolution trajectory checkpoint:

python scripts/infer.py \
  --checkpoint checkpoints/VGGT-Diff-Trajectory-480x832/vggt-diff.safetensors \
  --omega-checkpoint /path/to/vggt_omega_1b_512.pt \
  --example examples/garden \
  --height 480 \
  --width 832 \
  --steps 50 \
  --seed 20263012 \
  --fps 24 \
  --output outputs/garden

Outputs are saved to outputs/garden/. The model predicts 80 frames once and keeps those raw PNGs in frames/; vggtdiff.mp4 appends the same frames in reverse order, producing a 160-frame, 24 fps forward-then-backward video without additional inference. Remove --height 480 --width 832 for the lower, faster default resolution. CPU offloading is enabled by default; use --vram-limit-gib 64 to retain more weights on a high-memory GPU or --vram-limit-gib -1 to disable offloading.

VGGT-Omega first estimates the six source cameras, then VGGT-Diff generates 80 novel views. Defaults are 832 × 480, 12 fps, and route 0 → 1 → 2 → 3 → 4 → 5; name inputs in physical walking order. The route interpolates camera centers and orientations without scene-collision checks.

python scripts/infer_pose_free.py \
  --checkpoint checkpoints/VGGT-Diff-Trajectory-480x832/vggt-diff.safetensors \
  --omega-checkpoint /path/to/vggt_omega_1b_512.pt \
  --source-dir /path/to/six_rgb_images \
  --output outputs/pose_free

No ground-truth poses or frames are read. Outputs are prediction.mp4, camera_trajectory.mp4, synchronized prediction_with_trajectory.mp4, 80 PNG frames, and cameras.json containing the estimated source cameras and target path used. Camera views use small antialiased frustums. Reorder the default path with --route-order 0 2 1 3 4 5.

For a custom 80-camera path, pass --trajectory-json /path/to/targets.json with exactly one of:

• target_w2c: 80 homogeneous 4 × 4 OpenCV world-to-camera matrices in the VGGT-Omega world saved in cameras.json.
• target_c2w_relative_to_source0: 80 homogeneous 4 × 4 camera-to-world matrices relative to source camera 0.

Optional target_intrinsics supplies 80 pixel-space 3 × 3 matrices; otherwise, the median of the six estimated source intrinsics is used. Keep custom paths collision-free and at a sensible distance from observed views.

Use this layout:

my_scene/
├── source_views/
│   ├── 00.png
│   ├── 01.png
│   └── ...
└── trajectory.json

trajectory.json contains source_w2c, target_w2c, source_intrinsics, and target_intrinsics. Extrinsics are OpenCV world-to-camera matrices; intrinsics are 3 × 3 pixel-space matrices at source-image resolution. Add intrinsics_image_size: [width, height] to make the loader reject an accidental image/intrinsics resolution mismatch.

python scripts/infer.py \
  --checkpoint checkpoints/VGGT-Diff-Trajectory-480x832/vggt-diff.safetensors \
  --omega-checkpoint /path/to/vggt_omega_1b_512.pt \
  --example /path/to/my_scene \
  --output outputs/my_scene

Training scenes follow the folder-form DL3DV convention:

dataset_root/
└── scene_id/
    ├── images_4/
    └── transforms.json

First cache the frozen VGGT-Omega source features:

python scripts/prepare_omega_cache.py \
  --dataset-root /path/to/dataset_root \
  --cache-root /path/to/omega_cache \
  --omega-checkpoint /path/to/vggt_omega_1b_512.pt

Both recipes freeze the Wan2.1 VAE and VGGT-Omega while training the DiT and conditioning modules. Their latent layouts differ, so their checkpoints are separate model families.

This is the independent-view protocol used for the quantitative results in the paper. Six sources and N targets produce 6 + N one-to-one image, camera, Plucker, and geometry slots without temporal VAE compression.

accelerate launch scripts/train.py \
  --training-mode nvs \
  --dataset-root /path/to/dataset_root \
  --omega-cache /path/to/omega_cache \
  --target-frames 1 2 4 \
  --height 192 --width 336 \
  --backbone-lr 1e-5 \
  --condition-lr 1e-4 \
  --output outputs/nvs_training_run

The paper progresses the half-resolution target curriculum through {1,2,4} -> {2,4,8} -> {4,8,12} -> {4,8,12,16}, then trains at 480x832 with {4,8} -> {4,8,12} -> {4,8,12,16} and a 5e-6 backbone learning rate. Resume the previous stage and change --target-frames, resolution, and learning rate accordingly.

An 80-frame target is padded to 81 frames and causally compressed into 21 target slots. Camera conditions bypass the VAE; the four ordered Plucker maps in each causal window are packed and projected by the temporal Plucker adapter.

accelerate launch scripts/train.py \
  --training-mode trajectory \
  --dataset-root /path/to/dataset_root \
  --omega-cache /path/to/omega_cache \
  --resume /path/to/vggtdiff_trajectory.safetensors \
  --target-frames 80 \
  --height 480 \
  --width 832 \
  --output outputs/trajectory_training_run

The paper's original trajectory adaptation uses 192x336; the command above selects the full-resolution continuation. Both recipes use source-anchor camera normalization and point-track residual consistency.

@article{chen2026vggtdiff,
  title={VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis},
  author={Chen, Kangjie and Li, Xiangyu and Zhang, Dongbin and Zheng, Chaoda and Chen, Shijia and Deng, Jinhao and Lin, Hongbin and Choo, Sin Wai and Wang, Minqi and Yang, Minghao and Zhong, Dake and Song, Guorui and Zhang, Yu and Liu, Xianming and Wang, Boyang},
  journal={arXiv preprint arXiv:2609.33253},
  year={2026}
}

This project builds on Wan2.1, VGGT-Omega, FrameCrafter, and DiffSynth-Studio. We thank their authors for releasing their work.

VGGT-Diff is released under the VGGT-Diff Research License for non-commercial research and educational use. Third-party components and dependencies remain subject to their respective licenses and terms; see NOTICE.