gradio tweaks

README.md

@@ -7,6 +7,284 @@ sdk: gradio
 sdk_version: 4.38.1
 app_file: webgui.py
 pinned: false
+suggested_hardware: a10g-large
 ---
+<h1 align='center'>EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning</h1>
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+<div align='center'>
+    <a href='https://github.com/yuange250' target='_blank'>Zhiyuan Chen</a><sup>*</sup>&emsp;
+    <a href='https://github.com/JoeFannie' target='_blank'>Jiajiong Cao</a><sup>*</sup>&emsp;
+    <a href='https://github.com/octavianChen' target='_blank'>Zhiquan Chen</a><sup></sup>&emsp;
+    <a href='https://github.com/lymhust' target='_blank'>Yuming Li</a><sup></sup>&emsp;
+    <a href='https://github.com/' target='_blank'>Chenguang Ma</a><sup></sup>
+</div>
+<div align='center'>
+    *Equal Contribution.
+</div>
+
+<div align='center'>
+Terminal Technology Department, Alipay, Ant Group.
+</div>
+<br>
+<div align='center'>
+    <a href='https://badtobest.github.io/echomimic.html'><img src='https://img.shields.io/badge/Project-Page-blue'></a>
+    <a href='https://huggingface.co/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a>
+    <a href='https://www.modelscope.cn/models/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/ModelScope-Model-purple'></a>
+    <a href='https://arxiv.org/abs/2407.08136'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a>
+    <a href='assets/echomimic.png'><img src='https://badges.aleen42.com/src/wechat.svg'></a>
+</div>
+
+## &#x1F4E3; &#x1F4E3; Updates
+* [2024.07.17] 🔥🔥🔥 Accelerated models and pipe are released. The inference speed can be improved by **10x** (from ~7mins/240frames to ~50s/240frames on V100 GPU)
+* [2024.07.14] 🔥 [ComfyUI](https://github.com/smthemex/ComfyUI_EchoMimic) is now available. Thanks @smthemex for the contribution. 
+* [2024.07.13] 🔥 Thanks [NewGenAI](https://www.youtube.com/@StableAIHub) for the [video installation tutorial](https://www.youtube.com/watch?v=8R0lTIY7tfI).
+* [2024.07.13] 🔥 We release our pose&audio driven codes and models.
+* [2024.07.12] 🔥 WebUI and GradioUI versions are released. We thank @greengerong @Robin021 and @O-O1024 for their contributions.
+* [2024.07.12] 🔥 Our [paper](https://arxiv.org/abs/2407.08136) is in public on arxiv.
+* [2024.07.09] 🔥 We release our audio driven codes and models.
+
+## Gallery
+### Audio Driven (Sing)
+
+<table class="center">
+    
+<tr>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/d014d921-9f94-4640-97ad-035b00effbfe" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/877603a5-a4f9-4486-a19f-8888422daf78" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/e0cb5afb-40a6-4365-84f8-cb2834c4cfe7" muted="false"></video>
+    </td>
+</tr>
+
+</table>
+
+### Audio Driven (English)
+
+<table class="center">
+    
+<tr>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/386982cd-3ff8-470d-a6d9-b621e112f8a5" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/5c60bb91-1776-434e-a720-8857a00b1501" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/1f15adc5-0f33-4afa-b96a-2011886a4a06" muted="false"></video>
+    </td>
+</tr>
+
+</table>
+
+### Audio Driven (Chinese)
+
+<table class="center">
+    
+<tr>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/a8092f9a-a5dc-4cd6-95be-1831afaccf00" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/c8b5c59f-0483-42ef-b3ee-4cffae6c7a52" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/532a3e60-2bac-4039-a06c-ff6bf06cb4a4" muted="false"></video>
+    </td>
+</tr>
+
+</table>
+
+### Landmark Driven
+
+<table class="center">
+    
+<tr>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/1da6c46f-4532-4375-a0dc-0a4d6fd30a39" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/d4f4d5c1-e228-463a-b383-27fb90ed6172" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/18bd2c93-319e-4d1c-8255-3f02ba717475" muted="false"></video>
+    </td>
+</tr>
+
+</table>
+
+### Audio + Selected Landmark Driven
+
+<table class="center">
+    
+<tr>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/4a29d735-ec1b-474d-b843-3ff0bdf85f55" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/b994c8f5-8dae-4dd8-870f-962b50dc091f" muted="false"></video>
+    </td>
+    <td width=30% style="border: none">
+        <video controls loop src="https://github.com/BadToBest/EchoMimic/assets/11451501/955c1d51-07b2-494d-ab93-895b9c43b896" muted="false"></video>
+    </td>
+</tr>
+
+</table>
+
+**（Some demo images above are sourced from image websites. If there is any infringement, we will immediately remove them and apologize.）**
+
+## Installation
+
+### Download the Codes
+
+```bash
+  git clone https://github.com/BadToBest/EchoMimic
+  cd EchoMimic
+```
+
+### Python Environment Setup
+
+- Tested System Environment: Centos 7.2/Ubuntu 22.04, Cuda >= 11.7
+- Tested GPUs: A100(80G) / RTX4090D (24G) / V100(16G)
+- Tested Python Version: 3.8 / 3.10 / 3.11
+
+Create conda environment (Recommended):
+
+```bash
+  conda create -n echomimic python=3.8
+  conda activate echomimic
+```
+
+Install packages with `pip`
+```bash
+  pip install -r requirements.txt
+```
+
+### Download ffmpeg-static
+Download and decompress [ffmpeg-static](https://www.johnvansickle.com/ffmpeg/old-releases/ffmpeg-4.4-amd64-static.tar.xz), then
+```
+export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static
+```
+
+### Download pretrained weights
+
+```shell
+git lfs install
+git clone https://huggingface.co/BadToBest/EchoMimic pretrained_weights
+```
+
+The **pretrained_weights** is organized as follows.
+
+```
+./pretrained_weights/
+├── denoising_unet.pth
+├── reference_unet.pth
+├── motion_module.pth
+├── face_locator.pth
+├── sd-vae-ft-mse
+│   └── ...
+├── sd-image-variations-diffusers
+│   └── ...
+└── audio_processor
+    └── whisper_tiny.pt
+```
+
+In which **denoising_unet.pth** / **reference_unet.pth** / **motion_module.pth** / **face_locator.pth** are the main checkpoints of **EchoMimic**. Other models in this hub can be also downloaded from it's original hub, thanks to their brilliant works:
+- [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)
+- [sd-image-variations-diffusers](https://huggingface.co/lambdalabs/sd-image-variations-diffusers)
+- [audio_processor(whisper)](https://openaipublic.azureedge.net/main/whisper/models/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt)
+
+### Audio-Drived Algo Inference 
+Run the python inference script:
+
+```bash
+  python -u infer_audio2vid.py
+  python -u infer_audio2vid_pose.py
+```
+
+### Audio-Drived Algo Inference On Your Own Cases 
+
+Edit the inference config file **./configs/prompts/animation.yaml**, and add your own case:
+
+```bash
+test_cases:
+  "path/to/your/image":
+    - "path/to/your/audio"
+```
+
+The run the python inference script:
+```bash
+  python -u infer_audio2vid.py
+```
+
+### Motion Alignment between Ref. Img. and Driven Vid.
+
+(Firstly download the checkpoints with '_pose.pth' postfix from huggingface)
+
+Edit driver_video and ref_image to your path in demo_motion_sync.py, then run
+```bash
+  python -u demo_motion_sync.py
+```
+
+### Audio&Pose-Drived Algo Inference
+Edit ./configs/prompts/animation_pose.yaml, then run
+```bash
+  python -u infer_audio2vid_pose.py
+```
+
+### Pose-Drived Algo Inference
+Set draw_mouse=True in line 135 of infer_audio2vid_pose.py. Edit ./configs/prompts/animation_pose.yaml, then run
+```bash
+  python -u infer_audio2vid_pose.py
+```
+
+### Run the Gradio UI
+
+Thanks to the contribution from @Robin021:
+
+```bash
+
+python -u webgui.py --server_port=3000
+
+```
+
+## Release Plans
+
+|  Status  | Milestone                                                                | ETA |
+|:--------:|:-------------------------------------------------------------------------|:--:|
+|    ✅    | The inference source code of the Audio-Driven algo meet everyone on GitHub   | 9th July, 2024 |
+|    ✅    | Pretrained models trained on English and Mandarin Chinese to be released | 9th July, 2024 |
+|    ✅    | The inference source code of the Pose-Driven algo meet everyone on GitHub   | 13th July, 2024 |
+|    ✅    | Pretrained models with better pose control to be released                | 13th July, 2024 |
+|    ✅    | Accelerated models to be released                                        | 17th July, 2024 |
+|    🚀    | Pretrained models with better sing performance to be released            | TBD |
+|    🚀    | Large-Scale and High-resolution Chinese-Based Talking Head Dataset       | TBD |
+
+## Acknowledgements
+
+We would like to thank the contributors to the [AnimateDiff](https://github.com/guoyww/AnimateDiff), [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone) and [MuseTalk](https://github.com/TMElyralab/MuseTalk) repositories, for their open research and exploration. 
+
+We are also grateful to [V-Express](https://github.com/tencent-ailab/V-Express) and [hallo](https://github.com/fudan-generative-vision/hallo) for their outstanding work in the area of diffusion-based talking heads.
+
+If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.
+
+## Citation
+
+If you find our work useful for your research, please consider citing the paper :
+
+```
+@misc{chen2024echomimic,
+  title={EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning},
+  author={Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, Chenguang Ma},
+  year={2024},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV}
+}
+```
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=BadToBest/EchoMimic&type=Date)](https://star-history.com/?spm=5176.28103460.0.0.342a3da23STWrU#BadToBest/EchoMimic&Date)

webgui.py

@@ -211,30 +211,67 @@ def process_video(uploaded_img, uploaded_audio, width, height, length, seed, fac
   
 with gr.Blocks() as demo:
     gr.Markdown('# EchoMimic')
-    gr.Markdown('![]()')
+    gr.Markdown('## Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning')
+    gr.Markdown('Inference time: from ~7mins/240frames to ~50s/240frames on V100 GPU')
     with gr.Row():
         with gr.Column():
             uploaded_img = gr.Image(type="filepath", label="Reference Image")
             uploaded_audio = gr.Audio(type="filepath", label="Input Audio")
+            with gr.Accordion("Advanced Configuration", open=False):
+                with gr.Row():
+                    width = gr.Slider(label="Width", minimum=128, maximum=1024, value=default_values["width"])
+                    height = gr.Slider(label="Height", minimum=128, maximum=1024, value=default_values["height"])
+                with gr.Row():
+                    length = gr.Slider(label="Length", minimum=100, maximum=5000, value=default_values["length"])
+                    seed = gr.Slider(label="Seed", minimum=0, maximum=10000, value=default_values["seed"])
+                with gr.Row():
+                    facemask_dilation_ratio = gr.Slider(label="Facemask Dilation Ratio", minimum=0.0, maximum=1.0, step=0.01, value=default_values["facemask_dilation_ratio"])
+                    facecrop_dilation_ratio = gr.Slider(label="Facecrop Dilation Ratio", minimum=0.0, maximum=1.0, step=0.01, value=default_values["facecrop_dilation_ratio"])
+                with gr.Row():
+                    context_frames = gr.Slider(label="Context Frames", minimum=0, maximum=50, step=1, value=default_values["context_frames"])
+                    context_overlap = gr.Slider(label="Context Overlap", minimum=0, maximum=10, step=1, value=default_values["context_overlap"])
+                with gr.Row():
+                    cfg = gr.Slider(label="CFG", minimum=0.0, maximum=10.0, step=0.1, value=default_values["cfg"])
+                    steps = gr.Slider(label="Steps", minimum=1, maximum=100, step=1, value=default_values["steps"])
+                with gr.Row():
+                    sample_rate = gr.Slider(label="Sample Rate", minimum=8000, maximum=48000, step=1000, value=default_values["sample_rate"])
+                    fps = gr.Slider(label="FPS", minimum=1, maximum=60, step=1, value=default_values["fps"])
+                    device = gr.Radio(label="Device", choices=["cuda", "cpu"], value=default_values["device"])
+            generate_button = gr.Button("Generate Video")
         with gr.Column():
             output_video = gr.Video()
-
-    with gr.Accordion("Configuration", open=False):
-        width = gr.Slider(label="Width", minimum=128, maximum=1024, value=default_values["width"])
-        height = gr.Slider(label="Height", minimum=128, maximum=1024, value=default_values["height"])
-        length = gr.Slider(label="Length", minimum=100, maximum=5000, value=default_values["length"])
-        seed = gr.Slider(label="Seed", minimum=0, maximum=10000, value=default_values["seed"])
-        facemask_dilation_ratio = gr.Slider(label="Facemask Dilation Ratio", minimum=0.0, maximum=1.0, step=0.01, value=default_values["facemask_dilation_ratio"])
-        facecrop_dilation_ratio = gr.Slider(label="Facecrop Dilation Ratio", minimum=0.0, maximum=1.0, step=0.01, value=default_values["facecrop_dilation_ratio"])
-        context_frames = gr.Slider(label="Context Frames", minimum=0, maximum=50, step=1, value=default_values["context_frames"])
-        context_overlap = gr.Slider(label="Context Overlap", minimum=0, maximum=10, step=1, value=default_values["context_overlap"])
-        cfg = gr.Slider(label="CFG", minimum=0.0, maximum=10.0, step=0.1, value=default_values["cfg"])
-        steps = gr.Slider(label="Steps", minimum=1, maximum=100, step=1, value=default_values["steps"])
-        sample_rate = gr.Slider(label="Sample Rate", minimum=8000, maximum=48000, step=1000, value=default_values["sample_rate"])
-        fps = gr.Slider(label="FPS", minimum=1, maximum=60, step=1, value=default_values["fps"])
-        device = gr.Radio(label="Device", choices=["cuda", "cpu"], value=default_values["device"])
-
-    generate_button = gr.Button("Generate Video")
+            gr.Examples(
+                label = "Portrait examples",
+                examples = [
+                    ['assets/test_imgs/a.png'],
+                    ['assets/test_imgs/b.png'],
+                    ['assets/test_imgs/c.png'],
+                    ['assets/test_imgs/d.png'],
+                    ['assets/test_imgs/e.png']
+                ],
+                inputs = [uploaded_img]
+            )
+            gr.Examples(
+                label = "Audio examples",
+                examples = [
+                    ['assets/test_audios/chunnuanhuakai.wav'],
+                    ['assets/test_audios/chunwang.wav'],
+                    ['assets/test_audios/echomimic_en_girl.wav'],
+                    ['assets/test_audios/echomimic_en.wav'],
+                    ['assets/test_audios/echomimic_girl.wav'],
+                    ['assets/test_audios/echomimic.wav'],
+                    ['assets/test_audios/jane.wav'],
+                    ['assets/test_audios/mei.wav'],
+                    ['assets/test_audios/walden.wav'],
+                    ['assets/test_audios/yun.wav'],
+                ],
+                inputs = [uploaded_audio]
+            )
+            gr.HTML("""
+            <a href="https://huggingface.co/spaces/fffiloni/EchoMimic?duplicate=true">
+                <img src="https://huggingface.co/datasets/huggingface/badges/resolve/main/duplicate-this-space-xl.svg" alt="Duplicate this Space">
+            </a>
+            """)
 
     def generate_video(uploaded_img, uploaded_audio, width, height, length, seed, facemask_dilation_ratio, facecrop_dilation_ratio, context_frames, context_overlap, cfg, steps, sample_rate, fps, device):
 
@@ -263,7 +300,8 @@ with gr.Blocks() as demo:
             fps,
             device
         ],
-        outputs=output_video
+        outputs=output_video,
+        show_api=False
     )
 parser = argparse.ArgumentParser(description='EchoMimic')
 parser.add_argument('--server_name', type=str, default='0.0.0.0', help='Server name')
@@ -273,5 +311,5 @@ args = parser.parse_args()
 # demo.launch(server_name=args.server_name, server_port=args.server_port, inbrowser=True)
 
 if __name__ == '__main__':
-    demo.launch()
+    demo.queue(max_size=3).launch(show_api=False, show_error=True)
     #demo.launch(server_name=args.server_name, server_port=args.server_port, inbrowser=True)