Update README.md

README.md

@@ -1,59 +1,73 @@
 ---
-license: llama3
-base_model: meta-llama/Meta-Llama-3-8B
+license: mit
+language:
+- en
+pipeline_tag: text-generation
 tags:
-- generated_from_trainer
-datasets:
-- customized
-model-index:
-- name: astrollama-3-8b_summary_lmflow
-  results: []
+- llama-3
+- astronomy
+- astrophysics
+- arxiv
+inference: false
+base_model:
+- meta-llama/Llama-3-8b-hf
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# AstroLLaMA-3-8B-Base_AIC
 
-# astrollama-3-8b_summary_lmflow
+AstroLLaMA-3-8B is a specialized base language model for astronomy, developed by fine-tuning Meta's LLaMA-3-8b architecture on astronomical literature. This model was developed by the AstroMLab team. It is designed for next token prediction tasks and is not an instruct/chat model.
 
-This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the customized dataset.
+## Model Details
 
-## Model description
+- **Base Architecture**: LLaMA-3-8b
+- **Training Data**: Abstract, Introduction, and Conclusion (AIC) sections from arXiv's astro-ph category papers (from arXiv's inception up to January 2024)
+- **Data Processing**: Optical character recognition (OCR) on PDF files using the Nougat tool, followed by summarization using Qwen-2-8B and LLaMA-3.1-8B.
+- **Fine-tuning Method**: Continual Pre-Training (CPT) using the LMFlow framework
+- **Training Details**:
+  - Learning rate: 2 × 10⁻⁵
+  - Total batch size: 96
+  - Maximum token length: 512
+  - Warmup ratio: 0.03
+  - No gradient accumulation
+  - BF16 format
+  - Cosine decay schedule for learning rate reduction
+  - Training duration: 1 epoch (approximately 32 A100 GPU hours)
+- **Primary Use**: Next token prediction for astronomy-related text generation and analysis
+- **Reference**: Pan et al. 2024 [Link to be added]
 
-More information needed
+## Generating text from a prompt
 
-## Intended uses & limitations
+[The code example remains the same as in the previous version]
 
-More information needed
+## Model Limitations and Biases
 
-## Training and evaluation data
+A key limitation identified during the development of this model is that training solely on astro-ph data may not be sufficient to significantly improve performance over the base model, especially for the already highly performant LLaMA-3 series. This suggests that to achieve substantial gains, future iterations may need to incorporate a broader range of high-quality astronomical data beyond arXiv, such as textbooks, Wikipedia, and curated summaries.
 
-More information needed
+Here's a performance comparison chart based upon the astronomical benchmarking Q&A as described in [Ting et al. 2024](https://arxiv.org/abs/2407.11194), and Pan et al. 2024:
 
-## Training procedure
+| Model | Score (%) |
+|-------|-----------|
+| **AstroLLaMA-3-8B (AstroMLab)** | **72.3** |
+| LLaMA-3-8B | 72.0 |
+| Gemma-2-9B | 71.5 |
+| Qwen-2.5-7B | 70.4 |
+| Yi-1.5-9B | 68.4 |
+| InternLM-2.5-7B | 64.0 |
+| Mistral-7B-v0.3 | 63.9 |
+| ChatGLM3-6B | 50.4 |
 
-### Training hyperparameters
+As shown, while AstroLLaMA-3-8B performs competitively among models in its class, it does not surpass the performance of the base LLaMA-3-8B model. This underscores the challenges in developing specialized models and the need for more diverse and comprehensive training data.
 
-The following hyperparameters were used during training:
-- learning_rate: 2e-05
-- train_batch_size: 24
-- eval_batch_size: 8
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 7
-- total_train_batch_size: 168
-- total_eval_batch_size: 56
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.03
-- num_epochs: 1.0
+It's worth noting that the AstroLLaMA-3-8B-Plus which we will release in the next model release addresses these limitations by expanding beyond astro-ph data.
 
-### Training results
+## Ethical Considerations
 
+While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications.
 
+## Citation
 
-### Framework versions
+If you use this model in your research, please cite:
 
-- Transformers 4.41.2
-- Pytorch 2.3.1+cu121
-- Datasets 2.14.6
-- Tokenizers 0.19.1
+```
+[Citation for Pan et al. 2024 to be added]
+```