Update README.md

README.md

@@ -3,31 +3,28 @@ license: apache-2.0
 base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
 tags:
 - generated_from_trainer
-model-index:
-- name: tinyllama-mixpretrain-quinoa-sciphi
-  results: []
+datasets:
+  - cerebras/SlimPajama-627B
+  - bigcode/starcoderdata
+  - monsoon-nlp/greenbeing-proteins
+  - SciPhi/textbooks-are-all-you-need-lite
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 # tinyllama-mixpretrain-quinoa-sciphi
 
-This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) on the None dataset.
-
-## Model description
-
-More information needed
+TinyLLaMA model with continued pretraining / full-model finetuning on amino acids and simulated science textbooks.
 
-## Intended uses & limitations
+The goal is to a create models which understand amino acid sequences and natural language descriptions or Q&A.
 
-More information needed
+Training data was shuffled with:
+- 50% amino acid sequences / proteins from the [GreenBeing](https://huggingface.co/datasets/monsoon-nlp/greenbeing-proteins) research dataset (mostly quinoa)
+- 50% textbook content from the [SciPhi](https://huggingface.co/datasets/SciPhi/textbooks-are-all-you-need-lite) training dataset
 
-## Training and evaluation data
+## Training procedure
 
-More information needed
+CoLab notebook: https://colab.research.google.com/drive/1dah43byt-T0HQC9eCigNbxSZ8aHu6s-W?usp=sharing
 
-## Training procedure
+To fit on an L4 GPU, it was necessary to use max_length=400 and train_batch_size=1
 
 ### Training hyperparameters