Weronika Stryj

privategeek24

weronika-stryj-a863721b2

AI & ML interests

Text classification, Sentiment Analysis, Finance Risk analysis, Text generation, Question answearing, Optimization LLM, Cyber risk analysis, Speech to text

Recent Activity

Reacted to singhsidhukuldeep's post with 👀 about 1 month ago

While Google's Transformer might have introduced "Attention is all you need," Microsoft and Tsinghua University are here with the DIFF Transformer, stating, "Sparse-Attention is all you need." The DIFF Transformer outperforms traditional Transformers in scaling properties, requiring only about 65% of the model size or training tokens to achieve comparable performance. The secret sauce? A differential attention mechanism that amplifies focus on relevant context while canceling out noise, leading to sparser and more effective attention patterns. How? - It uses two separate softmax attention maps and subtracts them. - It employs a learnable scalar λ for balancing the attention maps. - It implements GroupNorm for each attention head independently. - It is compatible with FlashAttention for efficient computation. What do you get? - Superior long-context modeling (up to 64K tokens). - Enhanced key information retrieval. - Reduced hallucination in question-answering and summarization tasks. - More robust in-context learning, less affected by prompt order. - Mitigation of activation outliers, opening doors for efficient quantization. Extensive experiments show DIFF Transformer's advantages across various tasks and model sizes, from 830M to 13.1B parameters. This innovative architecture could be a game-changer for the next generation of LLMs. What are your thoughts on DIFF Transformer's potential impact?

Reacted to davidberenstein1957's post with 👍 about 1 month ago

Don't use an LLM when you can use a much cheaper model. The problem is that no one tells you how to actually do it. Just picking a pre-trained model (e.g., BERT) and throwing it at your problem won't work! If you want a small model to perform well on your problem, you need to fine-tune it. And to fine-tune it, you need data. The good news is that you don't need a lot of data but instead high-quality data for your specific problem. In the latest livestream, I showed you guys how to get started with Argilla on the Hub! Hope to see you at the next one. https://www.youtube.com/watch?v=BEe7shiG3rY

updated a collection about 2 months ago

Speech transcription

View all activity

Organizations

None yet

privategeek24's activity

upvoted 4 articles 3 months ago

Article

Llama 3.1 - 405B, 70B & 8B with multilinguality and long context

Jul 23

• 218

Article

TGI Multi-LoRA: Deploy Once, Serve 30 Models

Jul 18

• 51

Article

The 5 Most Under-Rated Tools on Hugging Face

Aug 22

• 85

Article

Memory-efficient Diffusion Transformers with Quanto and Diffusers

Jul 30

• 59

upvoted 2 papers 3 months ago

Rotary Position Embedding for Vision Transformer

Paper • 2403.13298 • Published Mar 20 • 3

RoFormer: Enhanced Transformer with Rotary Position Embedding

Paper • 2104.09864 • Published Apr 20, 2021 • 10

upvoted 2 articles 3 months ago

Article

Using LoRA for Efficient Stable Diffusion Fine-Tuning

Jan 26, 2023

• 39

Article

SetFit: Efficient Few-Shot Learning Without Prompts

Sep 26, 2022

• 19

upvoted a collection 3 months ago

PDF Document / OCR Datasets

Collection

Document datasets with .pdf files that are usable with pixparse libraries and tools. • 2 items • Updated Mar 30 • 47

upvoted an article 3 months ago

Article

Introducing TextImage Augmentation for Document Images

Aug 6

• 32