Update README.md

README.md

@@ -1,58 +1,108 @@
 ---
-base_model: output_models/abot-70b_epoch-1
+license: mit
+language:
+- en
+pipeline_tag: text-generation
 tags:
-- generated_from_trainer
-datasets:
-- customized
-model-index:
-- name: abot-70b_chat
-  results: []
+- llama-2
+- astronomy
+- astrophysics
+- arxiv
+inference: false
+base_model:
+- meta-llama/Llama-2-70b-hf
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# abot-70b_chat
-
-This model is a fine-tuned version of [output_models/abot-70b_epoch-1](https://huggingface.co/output_models/abot-70b_epoch-1) on the customized dataset.
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 2e-05
-- train_batch_size: 6
-- eval_batch_size: 1
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 8
-- total_train_batch_size: 48
-- total_eval_batch_size: 8
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.03
-- num_epochs: 1.0
-
-### Training results
-
-
-
-### Framework versions
-
-- Transformers 4.33.3
-- Pytorch 2.1.2+cu121
-- Datasets 2.14.6
-- Tokenizers 0.13.3
+# AstroLLaMA-2-70B-Chat_AIC
+
+AstroLLaMA-2-70B-Chat_AIC is a specialized chat model for astronomy, developed by fine-tuning the AstroLLaMA-2-70B-Base_AIC model. This model was developed by the AstroMLab team and is, to our best knowledge, one of the first specialized 70B parameter-level LLMs in astronomy designed for instruction-following and chat-based interactions.
+
+## Model Details
+
+- **Base Architecture**: LLaMA-2-70b
+- **Base Model**: AstroLLaMA-2-70B-Base_AIC (trained on Abstract, Introduction, and Conclusion sections from arXiv's astro-ph category papers)
+- **Fine-tuning Method**: Supervised Fine-Tuning (SFT)
+- **SFT Dataset**: 
+  - 10,356 astronomy-centered conversations generated from arXiv abstracts by GPT-4
+  - Full content of LIMA dataset
+  - 10,000 samples from Open Orca dataset
+  - 10,000 samples from UltraChat dataset
+- **Training Details**:
+  - Learning rate: 3 × 10⁻⁷
+  - Training epochs: 1
+  - Total batch size: 48
+  - Maximum token length: 2048
+  - Warmup ratio: 0.03
+  - Cosine decay schedule for learning rate reduction
+- **Primary Use**: Instruction-following and chat-based interactions for astronomy-related queries
+- **Reference**: Pan et al. 2024 [Link to be added]
+
+## Using the model for chat
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+# Load the model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("AstroMLab/astrollama-2-70b-chat_aic")
+model = AutoModelForCausalLM.from_pretrained("AstroMLab/astrollama-2-70b-chat_aic", device_map="auto")
+
+# Function to generate a response
+def generate_response(prompt, max_length=512):
+    full_prompt = f"###Human: {prompt}\n\n###Assistant:"
+    inputs = tokenizer(full_prompt, return_tensors="pt", truncation=True, max_length=max_length)
+    inputs = inputs.to(model.device)
+    
+    # Generate a response
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs, 
+            max_length=max_length, 
+            num_return_sequences=1, 
+            do_sample=True,
+            pad_token_id=tokenizer.eos_token_id,
+            eos_token_id=tokenizer.encode("###Human:", add_special_tokens=False)[0]
+        )
+    
+    # Decode and return the response
+    response = tokenizer.decode(outputs[0], skip_special_tokens=False)
+    
+    # Extract only the Assistant's response
+    assistant_response = response.split("###Assistant:")[-1].strip()
+    return assistant_response
+
+# Example usage
+user_input = "What are the main components of a galaxy?"
+response = generate_response(user_input)
+print(f"Human: {user_input}")
+print(f"Assistant: {response}")
+```
+
+## Model Performance and Limitations
+
+While the AstroLLaMA-2-70B-Base_AIC model demonstrated significant improvements over its baseline LLaMA-2-70B model, the chat version (AstroLLaMA-2-70B-Chat_AIC) experiences performance degradation due to limitations in the SFT process. Here's a performance comparison:
+
+
+Key limitations:
+
+1. **SFT Dataset Limitations**: The current SFT dataset, with only 30,000 Q&As (many not astronomy-focused), has proven inadequate for maintaining the base model's performance.
+2. **Performance Degradation**: The chat model's performance (64.7%) is significantly lower than the base model (76.0%), indicating an 11.3-point decrement due to the SFT process.
+3. **General Knowledge vs. Specialized Knowledge**: The current SFT process appears to deviate the model towards general answers, potentially at the cost of specialized astronomical knowledge.
+
+These limitations underscore the challenges in developing specialized chat models and the critical importance of both the quantity and quality of training data, especially for the SFT process.
+
+This model is released primarily for reproducibility purposes, allowing researchers to track the development process and compare different iterations of AstroLLaMA models.
+
+For optimal performance and the most up-to-date capabilities in astronomy-related tasks, we recommend using AstroSage-8B, where these limitations have been addressed through expanded training data and refined fine-tuning processes.
+
+## Ethical Considerations
+
+While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications.
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```
+[Citation for Pan et al. 2024 to be added]
+```