Spaces:
Running
on
Zero
Running
on
Zero
Stanislaw Szymanowicz
commited on
Commit
•
15f992e
1
Parent(s):
11ffd30
Add examples
Browse files- app.py +58 -12
- demo_examples/01_bigmac.png +0 -0
- demo_examples/02_hydrant.jpg +0 -0
- demo_examples/03_spyro.png +0 -0
- demo_examples/04_lysol.png +0 -0
- demo_examples/05_pinapple_bottle.png +0 -0
- demo_examples/06_unsplash_broccoli.png +0 -0
- demo_examples/07_objaverse_backpack.png +0 -0
- demo_examples/08_unsplash_chocolatecake.png +0 -0
- demo_examples/09_realfusion_cherry.png +0 -0
- demo_examples/10_triposr_teapot.png +0 -0
app.py
CHANGED
@@ -95,22 +95,23 @@ def main():
|
|
95 |
|
96 |
return ply_out_path
|
97 |
|
98 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
99 |
gr.Markdown(
|
100 |
"""
|
|
|
101 |
|
102 |
-
|
103 |
-
[Splatter Image](https://github.com/szymanowiczs/splatter-image) (CVPR 2024) is a fast, super cheap to train method for object 3D reconstruction from a single image.
|
104 |
The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
|
105 |
-
|
106 |
-
Upload an image of an object to see how the Splatter Image does.
|
107 |
-
|
108 |
-
**Comments:**
|
109 |
-
1. The first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
|
110 |
-
2. The model does not work well on photos of humans.
|
111 |
-
3. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. Artefacts might show - see video for more faithful results.
|
112 |
-
4. Best results are achieved on the datasets described in the [repository](https://github.com/szymanowiczs/splatter-image) using that code. This demo is experimental.
|
113 |
-
5. Our model might not be better than some state-of-the-art methods, but it is of comparable quality and is **much** cheaper to train and run.
|
114 |
"""
|
115 |
)
|
116 |
with gr.Row(variant="panel"):
|
@@ -131,6 +132,26 @@ def main():
|
|
131 |
)
|
132 |
with gr.Row():
|
133 |
submit = gr.Button("Generate", elem_id="generate", variant="primary")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
134 |
with gr.Column():
|
135 |
with gr.Row():
|
136 |
with gr.Tab("Reconstruction"):
|
@@ -140,6 +161,31 @@ def main():
|
|
140 |
interactive=False
|
141 |
)
|
142 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
143 |
submit.click(fn=check_input_image, inputs=[input_image]).success(
|
144 |
fn=preprocess,
|
145 |
inputs=[input_image, preprocess_background],
|
|
|
95 |
|
96 |
return ply_out_path
|
97 |
|
98 |
+
css = """
|
99 |
+
h1 {
|
100 |
+
text-align: center;
|
101 |
+
display:block;
|
102 |
+
}
|
103 |
+
"""
|
104 |
+
|
105 |
+
with gr.Blocks(css=css) as demo:
|
106 |
gr.Markdown(
|
107 |
"""
|
108 |
+
# Splatter Image
|
109 |
|
110 |
+
**Splatter Image (CVPR 2024)** [[code](https://github.com/szymanowiczs/splatter-image), [project page](https://szymanowiczs.github.io/splatter-image)] is a fast, super cheap-to-train method for object 3D reconstruction from a single image.
|
|
|
111 |
The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
|
112 |
+
Locally, on an NVIDIA V100 GPU, reconstruction (forward pass of the network) can be done at 38FPS and rendering (with Gaussian Splatting) at 588FPS.
|
113 |
+
Upload an image of an object or click on one of the provided examples to see how the Splatter Image does.
|
114 |
+
For best results clone the [main repository](https://github.com/szymanowiczs/splatter-image) and run the demo locally.
|
|
|
|
|
|
|
|
|
|
|
|
|
115 |
"""
|
116 |
)
|
117 |
with gr.Row(variant="panel"):
|
|
|
132 |
)
|
133 |
with gr.Row():
|
134 |
submit = gr.Button("Generate", elem_id="generate", variant="primary")
|
135 |
+
|
136 |
+
with gr.Row(variant="panel"):
|
137 |
+
gr.Examples(
|
138 |
+
examples=[
|
139 |
+
'./demo_examples/01_bigmac.png',
|
140 |
+
'./demo_examples/02_hydrant.jpg',
|
141 |
+
'./demo_examples/03_spyro.png',
|
142 |
+
'./demo_examples/04_lysol.png',
|
143 |
+
'./demo_examples/05_pinapple_bottle.png',
|
144 |
+
'./demo_examples/06_unsplash_broccoli.png',
|
145 |
+
'./demo_examples/07_objaverse_backpack.png',
|
146 |
+
'./demo_examples/08_unsplash_chocolatecake.png',
|
147 |
+
'./demo_examples/09_realfusion_cherry.png',
|
148 |
+
'./demo_examples/10_triposr_teapot.png'
|
149 |
+
],
|
150 |
+
inputs=[input_image],
|
151 |
+
cache_examples=False,
|
152 |
+
label="Examples",
|
153 |
+
examples_per_page=20,
|
154 |
+
)
|
155 |
with gr.Column():
|
156 |
with gr.Row():
|
157 |
with gr.Tab("Reconstruction"):
|
|
|
161 |
interactive=False
|
162 |
)
|
163 |
|
164 |
+
gr.Markdown(
|
165 |
+
"""
|
166 |
+
## Comments:
|
167 |
+
1. If you run the demo online, the first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
|
168 |
+
2. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. This is only an approximations and artefacts might show.
|
169 |
+
3. Known limitations include:
|
170 |
+
- sphere-like artefacts on the object and white halo around it: this is due to how the .ply mesh is extracted and limitations of the Gradio viewer
|
171 |
+
- see-through parts of objects, especially on the back: this is due to the model performing less well on more complicated shapes
|
172 |
+
- back of objects are blurry: this is a model limiation due to it being deterministic
|
173 |
+
4. Our model is of comparable quality to state-of-the-art methods, and is **much** cheaper to train and run.
|
174 |
+
|
175 |
+
## How does it work?
|
176 |
+
|
177 |
+
Splatter Image formulates 3D reconstruction as an image-to-image translation task. It maps the input image to another image,
|
178 |
+
in which every pixel represents one 3D Gaussian and the channels of the output represent parameters of these Gaussians, including their shapes, colours and locations.
|
179 |
+
The resulting image thus represents a set of Gaussians (almost like a point cloud) which reconstruct the shape and colour of the object.
|
180 |
+
The method is very cheap: the reconstruction amounts to a single forward pass of a neural network with only 2D operators (2D convolutions and attention).
|
181 |
+
The rendering is also very fast, due to using Gaussian Splatting.
|
182 |
+
Combined, this results in very cheap training and high-quality results.
|
183 |
+
For more results see the [project page](https://szymanowiczs.github.io/splatter-image) and the [CVPR article](https://arxiv.org/abs/2312.13150).
|
184 |
+
"""
|
185 |
+
)
|
186 |
+
|
187 |
+
|
188 |
+
|
189 |
submit.click(fn=check_input_image, inputs=[input_image]).success(
|
190 |
fn=preprocess,
|
191 |
inputs=[input_image, preprocess_background],
|
demo_examples/01_bigmac.png
ADDED
demo_examples/02_hydrant.jpg
ADDED
demo_examples/03_spyro.png
ADDED
demo_examples/04_lysol.png
ADDED
demo_examples/05_pinapple_bottle.png
ADDED
demo_examples/06_unsplash_broccoli.png
ADDED
demo_examples/07_objaverse_backpack.png
ADDED
demo_examples/08_unsplash_chocolatecake.png
ADDED
demo_examples/09_realfusion_cherry.png
ADDED
demo_examples/10_triposr_teapot.png
ADDED