Sylvain Delahaies's picture

3 2

Sylvain Delahaies

sylvain471

·

https://sdelahaies.fr

sylvain471

AI & ML interests

None yet

Recent Activity

reacted to Jaward's post with 👍 5 days ago

minimal single script implementation of knowledge distillation in LLMs. In this implementation, we use GPT-2 (124M) as student model and GPT-2 Medium (340M) as teacher via reverse Kullback-Leibler (KL) divergence, trained on a small chunk of openwebtext. Code: https://github.com/Jaykef/ai-algorithms/blob/main/llm_knowledge_distillation.ipynb

upvoted an article 7 days ago

The LASER technique: Evaluating SVD compression

new activity 8 days ago

HuggingFaceTB/SmolLM-135M:SmolLm and mergekit_moe: is lm_head missing ?

View all activity

Organizations

sylvain471's activity

New activity in HuggingFaceTB/SmolLM-135M 8 days ago

SmolLm and mergekit_moe: is lm_head missing ?

#14 opened 5 months ago by

New activity in sylvain471/climate_qas_fr 3 months ago

Librarian Bot: Add language metadata for dataset

#2 opened 3 months ago by

[bot] Conversion to Parquet

#1 opened 3 months ago by

parquet-converter

New activity in HuggingFaceTB/SmolLM-135M 5 months ago

SmolLm and mergekit_moe: is lm_head missing ?

#14 opened 5 months ago by

SmolLm and mergekit_moe: is lm_head missing ?

#14 opened 5 months ago by