Upload README.md

README.md

@@ -1,46 +1,94 @@
----
-license: apache-2.0
-pipeline_tag: text-to-audio
-tags:
-- text-to-music
-- music-generation
----
-
-# InspireMusic
-
+[//]: # (# InspireMusic)
 <p align="center">
  <a href="https://github.com/FunAudioLLM/InspireMusic" target="_blank">
-        <img alt="InspireMusic" src="https://svg-banners.vercel.app/api?type=origin&text1=Inspire%20Music🎶&text2=🤗%20A%20Fundamental%20Music%20Song%20Audio%20Generation%20Toolkit&width=800&height=210"></a>
+        <img alt="logo" src="./asset/logo.png" width="100%"></a>
 </p>
 
-[**Code**](https://inspiremusic.github.io/) | [**Demo**](https://iris2c.github.io/InspireMusic/) | [**ModelScope**](https://modelscope.cn/models/iic/InspireMusic/summary) | [**HuggingFace**](https://huggingface.co/FunAudioLLM/InspireMusic-Base)
+[//]: # (<p align="center">)
 
-InspireMusic is a fundamental AIGC toolkit designed for music, song, and audio generation using the PyTorch library. 
+[//]: # ( <a href="https://github.com/FunAudioLLM/InspireMusic" target="_blank">)
 
-## Introduction
-> [!Note]
-> This repo contains the algorithm infrastructure and some simple examples.
+[//]: # (        <img alt="InspireMusic" src="https://svg-banners.vercel.app/api?type=origin&text1=Inspire%20Music🎶&text2=🤗%20A%20Fundamental%20Music%20Song%20Audio%20Generation%20Toolkit&width=800&height=210"></a>)
 
-> [!Tip]
-> To explore the performance, please refer to [**InspireMusic Demo Page**](https://iris2c.github.io/InspireMusic). Space will also coming soon.
+[//]: # (</p>)
 
-InspireMusic is a unified music, song and audio generation framework through the audio tokenization and detokenization process integrated with a large autoregressive transformer. The original motive of this toolkit is to empower the common users to innovate soundscapes and enhance euphony in research through music, song, and audio crafting. The toolkit provides both inference and training code for AI generative models that create high-quality music. Featuring a unified framework, InspireMusic incorporates autoregressive Transformer and conditional flow-matching modeling (CFM), allowing for the controllable generation of music, songs, and audio with both textual and structural music conditioning, as well as neural audio tokenizers. Currently, the toolkit supports text-to-music generation and plans to expand its capabilities to include text-to-song and text-to-audio generation in the future.
+<p align="center">
+ <a href="https://iris2c.github.io/InspireMusic" target="_blank">
+        <img alt="Demo" src="https://img.shields.io/badge/Demo%20👈🏻-InspireMusic?labelColor=%20%23FDB062&label=InspireMusic&color=%20%23f79009"></a>
+<a href="https://github.com/FunAudioLLM/InspireMusic" target="_blank">
+        <img alt="Code" src="https://img.shields.io/badge/Code%20⭐-InspireMusic?labelColor=%20%237372EB&label=InspireMusic&color=%20%235462eb"></a>
+
+<a href="https://huggingface.co/FunAudioLLM/InspireMusic-Base" target="_blank">
+        <img alt="Model" src="https://img.shields.io/badge/InspireMusic-Model-lightgrey"></a>
+
+<a href="https://arxiv.org/abs/" target="_blank">
+        <img alt="Paper" src="https://img.shields.io/badge/arXiv-Paper-lightgrey"></a>
+<a href="https://github.com/FunAudioLLM/InspireMusic" target="_blank">
+
+[//]: # (<a href="https://huggingface.co/FunAudioLLM/InspireMusic-Base" target="_blank">)
+
+[//]: # (        <img alt="Model" src="https://img.shields.io/badge/Model-InspireMusic?labelColor=%20%23FDA199&label=InspireMusic&color=orange"></a>)
+
+[//]: # (<a href="https://arxiv.org/abs/" target="_blank">)
+
+[//]: # (        <img alt="Paper" src="https://img.shields.io/badge/Paper-arXiv?labelColor=%20%23528bff&label=arXiv&color=%20%23155EEF"></a>)
+
+[//]: # (<a href="https://github.com/FunAudioLLM/InspireMusic" target="_blank">)
+
+[//]: # (        <img alt="Githube Star" src="https://img.shields.io/github/stars/FunAudioLLM/InspireMusic"></a>)
+
+[//]: # (<a href="https://github.com/FunAudioLLM/InspireMusic/blob/main/asset/QR.jpg" target="_blank">)
+
+[//]: # (        <img src="https://img.shields.io/badge/group%20chat-group?&labelColor=%20%235462eb&color=%20%235462eb" alt="chat on WeChat"></a>)
+[//]: # (<a href="https://discord.gg/nSPpRU7fRr" target="_blank">)
+
+[//]: # (        <img src="https://img.shields.io/badge/discord-chat?&labelColor=%20%235462eb&color=%20%235462eb" alt="chat on Discord"></a>)
+
+[//]: # (    <a href="https://github.com/FunAudioLLM/InspireMusic" target="_blank">)
+
+[//]: # (        <img alt="Static Badge" src="https://img.shields.io/badge/v0.1-version?logo=free&color=%20%23155EEF&label=version&labelColor=%20%23528bff"></a>)
+[//]: # (<a href="https://github.com/FunAudioLLM/InspireMusic/graphs/commit-activity" target="_blank">)
+
+[//]: # (<img alt="Commits last month" src="https://img.shields.io/github/commit-activity/m/FunAudioLLM/InspireMusic?labelColor=%20%2332b583&color=%20%2312b76a"></a>)
+
+[//]: # (    <a href="https://github.com/FunAudioLLM/InspireMusic" target="_blank">)
 
+[//]: # (        <img alt="Issues closed" src="https://img.shields.io/github/issues-search?query=repo%3AFunAudioLLM%2FInspireMusic%20is%3Aclosed&label=issues%20closed&labelColor=%20%237d89b0&color=%20%235d6b98"></a>)
+
+[//]: # (    <a href="https://github.com/FunAudioLLM/InspireMusic/discussions/" target="_blank">)
+
+[//]: # (        <img alt="Discussion posts" src="https://img.shields.io/github/discussions/FunAudioLLM/InspireMusic?labelColor=%20%239b8afb&color=%20%237a5af8"></a>)
+</p>
+
+InspireMusic is a fundamental AIGC toolkit designed for music, song, and audio generation using the PyTorch library. 
+
+![GitHub Repo stars](https://img.shields.io/github/stars/FunAudioLLM/InspireMusic) Please support our community project 💖 by starring it on GitHub 加⭐支持 🙏 
+
+---
 <a name="Highligts"></a>
 ## Highlights
 **InspireMusic** focuses on music generation, song generation and audio generation.
 - A unified framework for music/song/audio generation.
 - Controllable with text prompts, music genres, music structures, etc.
 - Convenient Fine-tuning and Inference: Provide convenient fine-tuning and inference scripts and strategies, allowing users to easily their music generation models.
-- High audio quality.
+- Support text-to-music, music continuation tasks with high audio quality.
 
 <a name="What's News"></a>
 ## What's New 🔥
 
 [//]: # (- 2024/12: The [InspireMusic-Base]&#40;https://huggingface.co/FunAudioLLM/InspireMusic-Base&#41; voice understanding model is open-sourced, which offers high quality, diverse text style, music structure, music genre control capability.  )
-- 2024/11: Welcome to preview 👉🏻 [**InspireMusic Demos**](https://iris2c.github.io/InspireMusic) 👈🏻, more features and models will comming soon. We're excited to share this with you and are working hard to bring even more features and models very soon. Your support and feedback mean a lot to us!
+- 2024/11: Welcome to preview 👉🏻 [**InspireMusic Demos**](https://iris2c.github.io/InspireMusic) 👈🏻. We're excited to share this with you and are working hard to bring even more features and models soon. Your support and feedback mean a lot to us!
 - 2024/11: We are thrilled to announce the open-sourcing of the **InspireMusic** [code repository](https://github.com/FunAudioLLM/InspireMusic) and [demos](https://iris2c.github.io/InspireMusic). **InspireMusic** is a unified framework for music, song, and audio generation, featuring capabilities such as text-to-music conversion, music structure, genre control, and timestamp management. InspireMusic stands out for its exceptional music generation and instruction-following abilities.
 
+## Introduction
+> [!Note]
+> This repo contains the algorithm infrastructure and some simple examples.
+
+> [!Tip]
+> To explore the performance, please refer to [InspireMusic Demo Page](https://iris2c.github.io/InspireMusic). We will open-source InspireMusic models and HuggingFace Space soon.
+
+InspireMusic is a unified music, song and audio generation framework through the audio tokenization and detokenization process integrated with a large autoregressive transformer. The original motive of this toolkit is to empower the common users to innovate soundscapes and enhance euphony in research through music, song, and audio crafting. The toolkit provides both inference and training code for AI generative models that create high-quality music. Featuring a unified framework, InspireMusic incorporates autoregressive Transformer and conditional flow-matching modeling (CFM), allowing for the controllable generation of music, songs, and audio with both textual and structural music conditioning, as well as neural audio tokenizers. Currently, the toolkit supports text-to-music generation and plans to expand its capabilities to include text-to-song and text-to-audio generation in the future.
+
 ## Installation
 
 ### Clone
@@ -106,12 +154,12 @@ git clone https://www.modelscope.cn/iic/InspireMusic.git pretrained_models/Inspi
 ```
 
 ### Available Models
-Currently, we open source the music generation models that only support 24KHz mono channel audio. 
+Currently, we open source the music generation models only that supporting 24KHz mono channel audio. 
 The table below presents the links to the ModelScope and Huggingface model hub. More models will be available soon.
 
 | Model name            | Model Links                                                                                                                                                                               | Remarks                                        |
 |------------------------------|-------------------------------------------------------|-------------|
-| InspireMusic-Base     | [![model](https://img.shields.io/badge/ModelScope-Model-orange.svg)](https://modelscope.cn/models/iic/InspireMusic/summary) [![model](https://img.shields.io/badge/HuggingFace-Model-orange.svg)](https://huggingface.co/FunAudioLLM/InspireMusic-Base) | Pre-trained Music Generation Model, 24kHz mono |
+| InspireMusic-Base     | [![model](https://img.shields.io/badge/ModelScope-Model-lightgrey.svg)](https://modelscope.cn/models/iic/InspireMusic/summary) [![model](https://img.shields.io/badge/HuggingFace-Model-lightgrey.svg)](https://huggingface.co/FunAudioLLM/InspireMusic-Base) | Pre-trained Music Generation Model, 24kHz mono |
 | InspireMusic-1.5B     | [![model](https://img.shields.io/badge/ModelScope-Model-lightgrey.svg)]() [![model](https://img.shields.io/badge/HuggingFace-Model-lightgrey.svg)]()                                      | Pre-trained Music Generation 1.5B Model, 24kHz mono       |
 | InspireSong-Base      | [![model](https://img.shields.io/badge/ModelScope-Model-lightgrey.svg)]() [![model](https://img.shields.io/badge/HuggingFace-Model-lightgrey.svg)]()                                      | Pre-trained Song Generation Base Model, 24kHz mono         |
 | InspireSong-1.5B      | [![model](https://img.shields.io/badge/ModelScope-Model-lightgrey.svg)]() [![model](https://img.shields.io/badge/HuggingFace-Model-lightgrey.svg)]()                                      | Pre-trained Song Generation 1.5B Model, 24kHz mono         |
@@ -181,10 +229,11 @@ cd InspireMusic/examples/music_generation/
 bash infer.sh
 ```
 
-Here is an example code to run inference with flow matching model.
+Here is an example code to run inference with normal mode, i.e., with flow matching model for text-to-music and music continuation tasks.
 ```sh
 pretrained_model_dir = "./pretrained_models/InspireMusic/"
-python inspiremusic/bin/inference.py --mode sft \
+for task in 'text-to-music' 'continuation'; do
+  python inspiremusic/bin/inference.py --task $task \
       --gpu 0 \
       --config conf/inspiremusic.yaml \
       --prompt_data data/test/parquet/data.list \
@@ -192,16 +241,18 @@ python inspiremusic/bin/inference.py --mode sft \
       --llm_model $pretrained_model_dir/llm.pt \
       --music_tokenizer $pretrained_model_dir/music_tokenizer \
       --wavtokenizer $pretrained_model_dir/wavtokenizer \
-      --result_dir `pwd`/exp/inspiremusic/sft_test \
+      --result_dir `pwd`/exp/inspiremusic/${task}_test \
       --chorus verse \
       --min_generate_audio_seconds 8 \
       --max_generate_audio_seconds 30 
+done
 ```
 
-Here is an example code to run inference without flow matching model.
+Here is an example code to run inference with fast mode, i.e., without flow matching model for text-to-music and music continuation tasks.
 ```sh
 pretrained_model_dir = "./pretrained_models/InspireMusic/"
-python inspiremusic/bin/inference.py --mode sft \
+for task in 'text-to-music' 'continuation'; do
+  python inspiremusic/bin/inference.py --task $task \
       --gpu 0 \
       --config conf/inspiremusic.yaml \
       --prompt_data data/test/parquet/data.list \
@@ -209,17 +260,44 @@ python inspiremusic/bin/inference.py --mode sft \
       --llm_model $pretrained_model_dir/llm.pt \
       --music_tokenizer $pretrained_model_dir/music_tokenizer \
       --wavtokenizer $pretrained_model_dir/wavtokenizer \
-      --no_flow_mode True \
-      --result_dir `pwd`/exp/inspiremusic/sft_test \
+      --result_dir `pwd`/exp/inspiremusic/${task}_test \
       --chorus verse \
+      --fast True \
       --min_generate_audio_seconds 8 \
       --max_generate_audio_seconds 30 
+done
 ```
 
+## Roadmap
+
+- [ ] 2024/12
+  - [ ] 75Hz InspireMusic base model for music generation
+  - [ ] Support song generation task
+    
+- [ ] 2025/01
+    - [ ] 75Hz InspireSong model for song generation
+
+- [ ] 2025/02
+    - [ ] Support audio generation task 
+    - [ ] 75Hz InspireMusic-1.5B model for music generation
+
+- [ ] 2025/03
+    - [ ] 75Hz InspireAudio model for music and audio generation
+
+- [ ] TBD
+
+    - [ ] 25Hz InspireMusic model
+    - [ ] Support 48kHz stereo audio
+    - [ ] Streaming inference mode support
+    - [ ] Support more instruction mode, multi-lingual instructions
+    - [ ] InspireSong trained with more multi-lingual data
+    - [ ] More...
+
+---
 ### Friend Links
 Checkout some awesome Github repositories from Speech Lab of Institute for Intelligent Computing, Alibaba Group.
 
-<p align="left">
+<p align="center">
 <a href="https://github.com/modelscope/ClearerVoice-Studio" target="_blank">
         <img alt="Demo" src="https://img.shields.io/badge/Repo | Space-ClearVoice?labelColor=&label=ClearVoice&color=green"></a>
 <a href="https://github.com/FunAudioLLM/CosyVoice" target="_blank">
@@ -236,10 +314,10 @@ Checkout some awesome Github repositories from Speech Lab of Institute for Intel
   <table>
     <tr>
       <td style="text-align:center;">
-        <a href="./asset/QR.jpg"><img alt="FunAudioLLM in DingTalk" src="https://img.shields.io/badge/DingTalk-FunAudioLLM?labelColor=lightblue&label=FunAudioLLM&color=blue"></a>
+        <a href="./asset/QR.jpg"><img alt="FunAudioLLM in DingTalk" src="https://img.shields.io/badge/FunAudioLLM-DingTalk-d9d9d9"></a>
       </td>
       <td style="text-align:center;">
-        <a href="./asset/QR.jpg"><img alt="InspireMusic in WeChat" src="https://img.shields.io/badge/WeChat-InspireMusic?labelColor=lightblue&label=InspireMusic&color=blue"></a>
+        <a href="./asset/QR.jpg"><img alt="InspireMusic in WeChat" src="https://img.shields.io/badge/InspireMusic-WeChat-d9d9d9"></a>
       </td>
     </tr>
     <tr>
@@ -255,5 +333,15 @@ Checkout some awesome Github repositories from Speech Lab of Institute for Intel
 * [Github Discussion](https://github.com/FunAudioLLM/InspireMusic/discussions). Best for sharing feedback and asking questions.
 * [GitHub Issues](https://github.com/FunAudioLLM/InspireMusic/issues). Best for bugs you encounter using InspireMusic, and feature proposals.
 
+## Acknowledge
+
+1. We borrowed a lot of code from [CosyVoice](https://github.com/FunAudioLLM/CosyVoice).
+3. We borrowed a lot of code from [WavTokenizer](https://github.com/jishengpeng/WavTokenizer).
+4. We borrowed a lot of code from [AcademiCodec](https://github.com/yangdongchao/AcademiCodec).
+5. We borrowed a lot of code from [FunASR](https://github.com/modelscope/FunASR).
+6. We borrowed a lot of code from [FunCodec](https://github.com/modelscope/FunCodec).
+7. We borrowed a lot of code from [Matcha-TTS](https://github.com/shivammehta25/Matcha-TTS).
+9. We borrowed a lot of code from [WeNet](https://github.com/wenet-e2e/wenet).
+
 ## Disclaimer
 The content provided above is for academic purposes only and is intended to demonstrate technical capabilities. Some examples are sourced from the internet. If any content infringes on your rights, please contact us to request its removal.