|
--- |
|
license: cc-by-nc-2.0 |
|
pipeline_tag: image-to-3d |
|
library_name: transformers |
|
--- |
|
# [ECCV 2024] VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models |
|
|
|
[Porject page](https://junlinhan.github.io/projects/vfusion3d.html), [Paper link](https://arxiv.org/abs/2403.12034) |
|
|
|
VFusion3D is a large, feed-forward 3D generative model trained with a small amount of 3D data and a large volume of synthetic multi-view data. It is the first work exploring scalable 3D generative/reconstruction models as a step towards a 3D foundation. |
|
|
|
[VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models](https://junlinhan.github.io/projects/vfusion3d.html)<br> |
|
[Junlin Han](https://junlinhan.github.io/), [Filippos Kokkinos](https://www.fkokkinos.com/), [Philip Torr](https://www.robots.ox.ac.uk/~phst/)<br> |
|
GenAI, Meta and TVG, University of Oxford<br> |
|
European Conference on Computer Vision (ECCV), 2024 |
|
|
|
|
|
## News |
|
|
|
- [08.08.2024] [HF Demo](https://huggingface.co/spaces/facebook/VFusion3D) is available, big thanks to [Jade Choghari](https://github.com/jadechoghari)'s help for making it possible. |
|
- [25.07.2024] Release weights and inference code for VFusion3D. |
|
|
|
|
|
|
|
## Quick Start |
|
|
|
Getting started with VFusion3D is super easy! 🤗 Here’s how you can use the model with Hugging Face: |
|
|
|
### Install Dependencies (Optional) |
|
|
|
Depending on your needs, you may want to enable specific features like mesh generation or video rendering. We've got you covered with these additional packages: |
|
|
|
```bash |
|
!pip --quiet install imageio[ffmpeg] PyMCubes trimesh rembg[gpu,cli] kiui |
|
``` |
|
|
|
### Load model directly |
|
```python |
|
import torch |
|
from transformers import AutoModel, AutoProcessor |
|
|
|
# load the model and processor |
|
model = AutoModel.from_pretrained("jadechoghari/vfusion3d", trust_remote_code=True) |
|
processor = AutoProcessor.from_pretrained("jadechoghari/vfusion3d") |
|
|
|
# download and preprocess the image |
|
import requests |
|
from PIL import Image |
|
from io import BytesIO |
|
|
|
image_url = 'https://sm.ign.com/ign_nordic/cover/a/avatar-gen/avatar-generations_prsz.jpg' |
|
response = requests.get(image_url) |
|
image = Image.open(BytesIO(response.content)) |
|
|
|
# preprocess the image and get the source camera |
|
image, source_camera = processor(image) |
|
|
|
|
|
# generate planes (default output) |
|
output_planes = model(image, source_camera) |
|
print("Planes shape:", output_planes.shape) |
|
|
|
# generate a 3D mesh |
|
output_planes, mesh_path = model(image, source_camera, export_mesh=True) |
|
print("Planes shape:", output_planes.shape) |
|
print("Mesh saved at:", mesh_path) |
|
|
|
# Generate a video |
|
output_planes, video_path = model(image, source_camera, export_video=True) |
|
print("Planes shape:", output_planes.shape) |
|
print("Video saved at:", video_path) |
|
|
|
``` |
|
- **Default (Planes):** By default, VFusion3D outputs planes—ideal for further 3D operations. |
|
- **Export Mesh:** Want a 3D mesh? Just set `export_mesh=True`, and you'll get a `.obj` file ready to roll. You can also customize the mesh resolution by adjusting the `mesh_size` parameter. |
|
- **Export Video:** Fancy a 3D video? Set `export_video=True`, and you'll receive a beautifully rendered video from multiple angles. You can tweak `render_size` and `fps` to get the video just right. |
|
|
|
Check out our [demo app](https://huggingface.co/spaces/facebook/VFusion3D) to see VFusion3D in action! 🤗 |
|
|
|
## Results and Comparisons |
|
|
|
### 3D Generation Results |
|
<img src='assets/gif1.gif' width=950> |
|
|
|
<img src='assets/gif2.gif' width=950> |
|
|
|
### User Study Results |
|
<img src='assets/user.png' width=950> |
|
|
|
|
|
|
|
## Acknowledgement |
|
|
|
- This inference code of VFusion3D heavily borrows from [OpenLRM](https://github.com/3DTopia/OpenLRM). |
|
|
|
## Citation |
|
|
|
If you find this work useful, please cite us: |
|
|
|
|
|
``` |
|
@article{han2024vfusion3d, |
|
title={VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models}, |
|
author={Junlin Han and Filippos Kokkinos and Philip Torr}, |
|
journal={European Conference on Computer Vision (ECCV)}, |
|
year={2024} |
|
} |
|
``` |
|
|
|
## License |
|
|
|
- The majority of VFusion3D is licensed under CC-BY-NC, however portions of the project are available under separate license terms: OpenLRM as a whole is licensed under the Apache License, Version 2.0, while certain components are covered by NVIDIA's proprietary license. |
|
- The model weights of VFusion3D is also licensed under CC-BY-NC. |
