Running
on
Zero
193
🪄
ACE-Chat
(Tongyi Lab) ACE: All-round Creator and Editor
George Duan PRO
cctuan
·
AI & ML interests
None yet
Recent Activity
liked
a Space
18 days ago
scepter-studio/ACE-Chat
liked
a Space
19 days ago
jdh-algo/JoyType
liked
a Space
about 1 month ago
H-Liu1997/TANGO
Organizations
cctuan's activity
liked
a
Space
18 days ago
liked
a
Space
19 days ago
liked
a
Space
about 1 month ago
Reacted to
singhsidhukuldeep's
post with 👀
about 1 month ago
Post
2159
While Google's Transformer might have introduced "Attention is all you need," Microsoft and Tsinghua University are here with the DIFF Transformer, stating, "Sparse-Attention is all you need."
The DIFF Transformer outperforms traditional Transformers in scaling properties, requiring only about 65% of the model size or training tokens to achieve comparable performance.
The secret sauce? A differential attention mechanism that amplifies focus on relevant context while canceling out noise, leading to sparser and more effective attention patterns.
How?
- It uses two separate softmax attention maps and subtracts them.
- It employs a learnable scalar λ for balancing the attention maps.
- It implements GroupNorm for each attention head independently.
- It is compatible with FlashAttention for efficient computation.
What do you get?
- Superior long-context modeling (up to 64K tokens).
- Enhanced key information retrieval.
- Reduced hallucination in question-answering and summarization tasks.
- More robust in-context learning, less affected by prompt order.
- Mitigation of activation outliers, opening doors for efficient quantization.
Extensive experiments show DIFF Transformer's advantages across various tasks and model sizes, from 830M to 13.1B parameters.
This innovative architecture could be a game-changer for the next generation of LLMs. What are your thoughts on DIFF Transformer's potential impact?
The DIFF Transformer outperforms traditional Transformers in scaling properties, requiring only about 65% of the model size or training tokens to achieve comparable performance.
The secret sauce? A differential attention mechanism that amplifies focus on relevant context while canceling out noise, leading to sparser and more effective attention patterns.
How?
- It uses two separate softmax attention maps and subtracts them.
- It employs a learnable scalar λ for balancing the attention maps.
- It implements GroupNorm for each attention head independently.
- It is compatible with FlashAttention for efficient computation.
What do you get?
- Superior long-context modeling (up to 64K tokens).
- Enhanced key information retrieval.
- Reduced hallucination in question-answering and summarization tasks.
- More robust in-context learning, less affected by prompt order.
- Mitigation of activation outliers, opening doors for efficient quantization.
Extensive experiments show DIFF Transformer's advantages across various tasks and model sizes, from 830M to 13.1B parameters.
This innovative architecture could be a game-changer for the next generation of LLMs. What are your thoughts on DIFF Transformer's potential impact?
- 1 reply
liked
a
Space
about 1 month ago
Reacted to
KingNish's
post with ❤️
2 months ago
Post
3082
A super good and fast image inpainting demo is here.
Its' super cool and realistic.
Demo by @OzzyGT (Must try):
OzzyGT/diffusers-fast-inpaint
Its' super cool and realistic.
Demo by @OzzyGT (Must try):
OzzyGT/diffusers-fast-inpaint
updated
2
models
2 months ago
liked
a
Space
3 months ago
liked
2
Spaces
4 months ago
Reacted to
MonsterMMORPG's
post with 🔥
4 months ago
Post
5261
Kling AI Video is FINALLY Public (All Countries), Free to Use and MIND BLOWING - Full Tutorial > https://youtu.be/zcpqAxYV1_w
You probably seen those mind blowing AI made videos. And the day has arrived. The famous Kling AI is now worldwide available for free. In this tutorial video I will show you how to register for free with just email to Kling AI and use its mind blowing text to video animation, image to video animation and text to image, and image to image capabilities. This video will show you non-cherry pick results so you will know the actual quality and capability of the model unlike those extremely cherry pick example demos. Still, #KlingAI is the only #AI model that competes with OpenAI's #SORA and it is real to use.
🔗 Kling AI Official Website ⤵️
▶️ https://www.klingai.com/
🔗 SECourses Discord Channel to Get Full Support ⤵️
▶️ https://discord.com/servers/software-engineering-courses-secourses-772774097734074388
🔗 Our GitHub Repository ⤵️
▶️ https://github.com/FurkanGozukara/Stable-Diffusion
🔗 Our Reddit ⤵️
▶️ https://www.reddit.com/r/SECourses/
You probably seen those mind blowing AI made videos. And the day has arrived. The famous Kling AI is now worldwide available for free. In this tutorial video I will show you how to register for free with just email to Kling AI and use its mind blowing text to video animation, image to video animation and text to image, and image to image capabilities. This video will show you non-cherry pick results so you will know the actual quality and capability of the model unlike those extremely cherry pick example demos. Still, #KlingAI is the only #AI model that competes with OpenAI's #SORA and it is real to use.
🔗 Kling AI Official Website ⤵️
▶️ https://www.klingai.com/
🔗 SECourses Discord Channel to Get Full Support ⤵️
▶️ https://discord.com/servers/software-engineering-courses-secourses-772774097734074388
🔗 Our GitHub Repository ⤵️
▶️ https://github.com/FurkanGozukara/Stable-Diffusion
🔗 Our Reddit ⤵️
▶️ https://www.reddit.com/r/SECourses/
- 5 replies
liked
a
model
4 months ago
liked
a
Space
6 months ago
updated
a
model
6 months ago