4DHumans: Reconstructing and Tracking Humans with Transformers

Code repository for the paper: Humans in 4D: Reconstructing and Tracking Humans with Transformers Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa^*, Jitendra Malik^* arXiv preprint 2023 [paper] [project page] [hugging faces space]

Download dependencies

Our demo code depends on detectron2 to detect humans. To automatically download this dependency, clone this repo using --recursive, or run git submodule update --init if you've already cloned the repository. You should see the detectron2 source code at vendor/detectron2.

git clone https://github.com/shubham-goel/4D-Humans.git --recursive
# OR
git clone https://github.com/shubham-goel/4D-Humans.git
cd 4D-Humans
git submodule update --init

Installation

We recommend creating a clean conda environment and installing all dependencies, as follows:

conda env create -f environment.yml

After the installation is complete you can activate the conda environment by running:

conda activate 4D-humans

Download checkpoints and SMPL models

To download the checkpoints and SMPL models, run

./fetch_data.sh

Run demo on images

You may now run our demo to 3D reconstruct humans in images using the following command, which will run ViTDet and HMR2.0 on all images in the specified --img_folder and save renderings of the reconstructions in --out_folder. You can also use the --side_view flag to additionally render the side view of the reconstructed mesh. --batch_size batches the images together for faster processing.

python demo.py \
    --img_folder example_data/images \
    --out_folder demo_out \
    --batch_size=48 --side_view

Run demo on videos

Coming soon.

Training and evaluation

Cmoing soon.

Acknowledgements

Parts of the code are taken or adapted from the following repos:

Additionally, we thank StabilityAI for a generous compute grant that enabled this work.

Citing

If you find this code useful for your research, please consider citing the following paper:

@article{4DHUMANS,
    title={Humans in 4{D}: Reconstructing and Tracking Humans with Transformers},
    author={Goel, Shubham and Pavlakos, Georgios and Rajasegaran, Jathushan and Kanazawa, Angjoo and Malik, Jitendra},
    journal={arXiv preprint},
    year={2023}
}