File size: 6,977 Bytes
645f2d9 8b78dfa 645f2d9 f81008b 645f2d9 36ea211 1905a93 36ea211 645f2d9 8028140 645f2d9 6f7b089 645f2d9 6f7b089 94356f2 6f7b089 645f2d9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 |
---
license: apache-2.0
---
<img src="https://cdn-uploads.huggingface.co/production/uploads/63118add64939fabc0108b28/BB42g4V8HTxb5dR4tcy8A.png" alt="DCLM Logo" width="300" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
# Model Card for DCLM-1B-IT
DCLM-IT-1B is a 1.4B billion parameter language model trained on the DCLM-Baseline dataset and then further finetuned on our DCLM-IT finetuning mixture. This model is designed to showcase the effectiveness of systematic data curation techniques for improving language model performance.
## Model Details
| Size | Training Tokens | Layers | Hidden Size | Attention Heads | Context Length |
|:------:|:-----------------:|:--------:|:-------------:|:-----------------:|:----------------:|
| 1.4B | 4.308T | 24 | 2048 | 16 | 2048 |
### Model Description
- **Developed by:** DataComp for Language Models (DCLM) Team
- **Model type:** Decoder-only Transformer language model
- **Language(s):** English (primarily)
- **License:** Apache 2.0
- **Contact:** contact@datacomp.ai
- **Date:** July 2024
### Model Sources
- **Repository:** https://github.com/mlfoundations/dclm
- **Dataset:** https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0
- **Paper:** [DataComp-LM: In search of the next generation of training sets for language models](https://arxiv.org/abs/2406.11794)
### Instruction Tuning Details
The model was trained using the following setup:
- **Architecture:** Decoder-only Transformer
- **Framework:** PyTorch with OpenLM
- **Optimizer:** AdamW
- **Learning Rate:** 2e-5 (peak)
- **Weight Decay:** 0.1
- **Batch Size:** 2048 sequences
- **Sequence Length:** 2048 tokens
- **Total Training Tokens:** 8.4B
- **Number of Epochs**: 10
- **Hardware:** Trained on H100 GPUs
For more detailed training information, please refer to Section 3.4 and Appendix F of the DCLM paper.
## Quickstart
First install open_lm
```
pip install git+https://github.com/mlfoundations/open_lm.git
```
Then you can load the model using HF's Auto classes as follows:
```python
from open_lm.hf import *
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("TRI-ML/DCLM-1B-IT")
model = AutoModelForCausalLM.from_pretrained("TRI-ML/DCLM-1B-IT")
inputs = tokenizer(["Machine learning is"], return_tensors="pt")
gen_kwargs = {"max_new_tokens": 50, "top_p": 0.8, "temperature": 0.8, "do_sample": True, "repetition_penalty": 1.1}
output = model.generate(inputs['input_ids'], **gen_kwargs)
output = tokenizer.decode(output[0].tolist(), skip_special_tokens=True)
print(output)
```
## Evaluation
Here are the evaluation results for DCLM-1B models on various tasks (using [llm-foundry](https://github.com/mosaicml/llm-foundry) eval suite)
| Task | Core | Extended | MMLU 5-shot |
|:---------:|:------:|:----------:|:-------------:|
| DCLM-1B | 45.2 | 28.1 | 47.5 |
| DCLM-1B-IT| 47.1 | 33.6 | 51.4 |
Moreover, we present our evaluation results on Length-Controlled Alpaca-Eval 2.0 to measure our instruction-following capabilities. We report results
from the leaderboard for non-DCLM models. We compare to state-of-the-art small models, and also include a few larger model sizes for comparison.
| Model | AlpacaEval2.0 LC Win-rate (%) |
|------------------------------------|------------------------------:|
| Qwen1.5 1.8B Chat | 2.6 |
| Gemma-Instruct-2B | 5.4 |
| Phi-2 SFT | 5.9 |
| DCLM-IT-1B | **8.6** |
| **Larger model sizes** | |
| Alpaca-7B | 5.9 |
| LLaMA-2-Chat-13B | 8.4 |
| DaVinci001 | 9.0 |
| Nous-Hermes-13B | 9.7 |
| Gemma-Instruct-7B | 10.4 |
| DCLM-IT-7B | 16.6 |
## Example Code
This is example code on how to run the chat model.
```
from transformers import AutoTokenizer
from open_lm.utils.transformers.hf_config import OpenLMConfig
import torch
from open_lm.utils.transformers.hf_model import OpenLMConfig, OpenLMforCausalLM
# Load the model and tokenizer
model_name = "TRI-ML/DCLM-1B-IT"
# Load the configuration, tokenizer, and model separately
config = OpenLMConfig.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = OpenLMforCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="cuda", config=config)
# Define the prompt format
def create_prompt(instruction):
PROMPT = '''Below is an instruction that describes a task.\n\nWrite a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:'''
return PROMPT.format(instruction=instruction)
# Example instruction
instruction = "Give me a poem about Sachin Tendulkar."
# Create the prompt
prompt = create_prompt(instruction)
# Tokenize the input
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(torch.device('cuda'))
# Generate the response
output = model.generate(input_ids, max_length=500, top_p=.95, do_sample=True, temperature=0.3)
# Decode the response
response = tokenizer.decode(output[0][len(input_ids[0]):])
response = response.split("<|endoftext|>")[0]
# Print the response
print(response)
```
## Limitations and Biases
While DCLM-1B demonstrates strong performance across a range of tasks, it's important to note:
1. The model may exhibit biases present in its training data, which is derived from web crawl data.
2. It has not undergone specific alignment or safety fine-tuning, so outputs should be used with caution.
3. Performance on tasks not included in the evaluation suite may vary.
4. The model's knowledge is limited to its training data cutoff date.
## Ethical Considerations
Users should be aware that this model, like all large language models, can potentially generate harmful or biased content. It should not be used for making decisions about individuals or in sensitive applications without appropriate safeguards and human oversight.
## Citation
If you use this model in your research, please cite:
```
@article{Li2024DataCompLM,
title={DataComp-LM: In search of the next generation of training sets for language models},
author={Jeffrey Li and Alex Fang and Georgios Smyrnis and Maor Ivgi and Matt Jordan and Samir Gadre and Hritik Bansal and Etash Guha and Sedrick Keh and Kushal Arora and [... full author list]},
journal={arXiv preprint arXiv:2406.11794},
year={2024}
}
``` |