Update README.md

README.md

@@ -10,31 +10,23 @@ datasets:
 # Model Summary
 <img src="llama-3-6B icon.jpeg" width="500" alt="Llama-3-6B"/>
 
-This is world's first Llama-3 base model with 6B params, it is a pretrained version of [prince-canuma/Llama-3-6B-v0](https://huggingface.co/prince-canuma/Llama-3-6B-v0) which was, downcycled from Meta-Llama-3-8B.
-It was continually pretrained on 1B tokens of enlish only text from fineweb and achieves the following results on the evaluation set:
+Introducing the world's first Llama-3 base model with 6B parameters. This model is a pretrained version of [prince-canuma/Llama-3-6B-v0](https://huggingface.co/prince-canuma/Llama-3-6B-v0), which was created from Meta-Llama-3-8B using a technique called [downcycling](https://youtube.com/playlist?list=PLDn_JsyofyfTH5_5V1MNb8UYKxMl6IMNy&si=9hcOol4KHIgWThgt) .
+The model was continually pretrained on 1 billion tokens of English-only text from fineweb, achieving impressive results on the evaluation set:
 - Loss: 2.4942
 
 <!-- Provide a longer summary of what this model is. -->
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
 ## Model Description
 
 <!-- Provide a longer summary of what this model is. -->
 This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
 
 - **Developed by:** [Prince Canuma](https://huggingface.co/prince-canuma)
-- **Model type:** Transformer
+- **Sponsored by:** General
+- **Model type:** Llama
+- **Language(s) (NLP):** [More Information Needed]
 - **License:** MIT
-- **Finetuned from model:** prince-canuma/Llama-3-6B-v0
+- **Pretrained from model:** prince-canuma/Llama-3-6B-v0
 
 ### Model Sources [optional]
 
@@ -89,6 +81,15 @@ Python 2 and Python 3 are two different versions of the Python language. Python
 
 ## Training Details
 
+### Downcycling
+
+A technique that allows you to create new LLMs of diversa sizes from checkpoints of large pretrained models.
+You take a reference model (i.e., Llama-3-8B) and copy the weights of 24 layers out of 32 layers alongside embedding and prediction heads. Then you initialize a smaller target model with 24 layers and load those pretrained weights.
+This new model will most likely still output legible outputs, but for it to perform well you need continue the pretraining.
+
+
+
+
 ### Training Data
 
 For continued pretrained, I extracted 1B tokens from [Huggingface's FineWeb CC-Main-2024-10](https://huggingface.co/datasets/HuggingFaceFW/fineweb#breakdown-by-dumpcrawl) slice.