SeG: Generative Semantic Gesture Retriever

Generative Semantic Gesture Retriever is a component designed to retrieve and generate semantic gestures based on input speech text. This repository provides all the necessary prepartions to set up and deploy the SeG model, including downloading the base model, merging finetuned LoRA model parameters, and deploying an API service.

Citation

If you use the models or codes, please cite our Paper.

@article{
  Zhang2024SemanticGesture,
  author = {Zhang, Zeyi and Ao, Tenglong and Zhang, Yuyao and Gao, Qingzhe and Lin, Chuan and Chen, Baoquan and Liu, Libin},
  title = {Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis},
  journal = {ACM Trans. Graph.},
  issue_date = {July 2024},
  numpages = {17},
  doi = {10.1145/3658134},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {co-speech gesture synthesis, multi-modality, retrieval augmentation}
}