File size: 7,451 Bytes
9df30db
 
02c68ec
 
 
 
 
 
 
 
728b3c4
0d57b8c
 
8e49fdb
0d57b8c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ac17d8d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
0d57b8c
 
 
 
 
 
 
 
b0e5599
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
0d57b8c
513cd61
 
ac17d8d
513cd61
0d57b8c
 
ac17d8d
0d57b8c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
d9bfbb2
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
---
license: cc-by-nc-4.0
datasets:
- tatsu-lab/alpaca
library_name: transformers
pipeline_tag: text-generation
tags:
- galactica
- alpaca
- opt
inference: false
---

# GALPACA 30B (large)

GALACTICA 30B fine-tuned on the Alpaca dataset.

The model card from the original Galactica repo can be found [here](https://github.com/paperswithcode/galai/blob/main/docs/model_card.md), and the original paper [here](https://galactica.org/paper.pdf).

The dataset card for Alpaca can be found [here](https://huggingface.co/datasets/tatsu-lab/alpaca/blob/main/README.md), and the project homepage [here](https://crfm.stanford.edu/2023/03/13/alpaca.html).
  The Alpaca dataset was collected with a modified version of the [Self-Instruct Framework](https://github.com/yizhongw/self-instruct), and was built using OpenAI's `text-davinci-003` model. As such it is subject to OpenAI's terms of service.

## Model Details

The GALACTICA models are trained on a large-scale scientific corpus and are designed to perform scientific tasks.
The Alpaca dataset is a set of 52k instruct-response pairs designed to enhace the instruction following capabilites of pre-trained language models.

## Model Use

The GALACTICA model card specifies that the primary indended users of the GALACTICA models are researchers studying language models applied to the scientific domain, and it cautions against production use of GALACTICA without safeguards due to the potential for the model to produce inaccurate information.
The original GALACTICA models are available under a non-commercial CC BY-NC 4.0 license, and the GALPACA model is additionally subject to the [OpenAI Terms of Service](https://openai.com/policies/terms-of-use).

<details>
  <summary> Example Outputs </summary>
  
*__Full Prompt:__*
```
Below is an instruction that describes a task. As a scientist, write a response that appropriately completes the request, citing specific papers whenever possible.

### Instruction:
Write out Maxwell's equations and explain each one.

### Markdown and LaTeX Response:
```

*__Response:__*

Maxwell's equations are a set of four equations that describe the behavior of electromagnetic fields. These equations are:
$$\nabla \times \textbf{E} = -\frac{\partial \textbf{B}}{\partial t}$$
$$\nabla \times \textbf{B} = \mu_0 \epsilon_0 \frac{\partial \textbf{E}}{\partial t} + \mu_0 \textbf{J}$$
$$\nabla \bullet \textbf{E} = \frac{\rho}{\epsilon_0}$$
$$\nabla \bullet \textbf{B} = 0$$
The first two equations describe how the electric and magnetic fields, E and B, are related to each other. The third and fourth equations are Gauss's law for electric and magnetic fields, respectively. These equations describe the behavior of the electric and magnetic fields when there are charges and currents in the space. Each equation has a specific purpose and together they form a complete description of the behavior of electromagnetic fields.

*__Full Prompt:__*
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Write a python function implements the Sherman-Morrison matrix inversion lemma using numpy.

### Response:
```python
```

*__Response:__*
```python
def sm_inv_lemma(A, u, v):
  """ Function implementing the Sherman-Morrison matrix inversion lemma using numpy.
  
  Parameters
  ----------
  A : numpy array
      The original matrix.
  
  u : numpy array
      The first vector in the Sherman-Morrison lemma.
  
  v : numpy array
      The second vector in the Sherman-Morrison lemma.
  
  Returns
  -------
  The inverse matrix of A - uv^T.
  """
  A_inv = np.linalg.inv(A)
  alpha = 1 / (1 + v.T @ A_inv @ u)
  return A_inv - (alpha * A_inv @ u @ v.T @ A_inv)
```

</details>

## Training Data

The GALPACA models are trained by fine-tuning pre-trained GALACTICA models on the Alpaca dataset. GALACTICA models were trained on 106 billion tokens of open-access scientific text and data, including papers, textbooks, scientific websites, encyclopedias, and more.
Fine-tuning the base GALACTICA models on the 52k instruction-response pairs in the Alpaca dataset allows users to query the GALPACA models in an instruct-response fashion.

## How to Use

The GALPACA weights are made available for use with the `transformers` library.

<details>
  <summary> Click to expand </summary>

```python
# pip install accelerate
from transformers import AutoTokenizer, OPTForCausalLM

tokenizer = AutoTokenizer.from_pretrained("GeorgiaTechResearchInstitute/galpaca-30b")
model = OPTForCausalLM.from_pretrained("GeorgiaTechResearchInstitute/galpaca-30b", device_map="auto", torch_dtype=torch.float16)

# see the original Alpaca repo for more information about the prompt templates
no_input_prompt_template = ("Below is an instruction that describes a task. "
                            "Write a response that appropriately completes the request.\n\n"
                            "### Instruction:\n{instruction}\n\n### Response:")
prompt = "Write out Maxwell's equations and explain the meaning of each one."
formatted_prompt = no_input_prompt_template.format_map({'instruction': prompt})

tokenized_prompt = tokenizer(formatted_prompt, return_tensors="pt").input_ids.to(model.device)
out_tokens = model.generate(tokenized_prompt)

print(tokenizer.batch_decode(out_tokens, skip_special_tokens=False, clean_up_tokenization_spaces=False))
```
</details>

## Training Resources

GALPACA 30B was fine-tuned in about 6 hours using 16 A100 80GB GPUS, 16-bit mixed-precision, an effective batch-size of 1024, and with a maximum context window of 384 tokens. This model was trained using DeepSpeed ZeRO Stage 3 optimizations.

## Performance and Limitations

Qualitative evaluation suggests that Galpaca frequently outperforms LLaMA-based Alpaca models on tasks related to technical knowledge and programming, while it underperforms on natural langauge tasks such as generating prose. More information about the performance and limitations of the GALACTICA family of models can be found on the original GALACTICA model card.

## Works Cited

```bibtex
@inproceedings{GALACTICA,
    title={GALACTICA: A Large Language Model for Science},
    author={Ross Taylor and Marcin Kardas and Guillem Cucurull and Thomas Scialom and Anthony Hartshorn and Elvis Saravia and Andrew Poulton and Viktor Kerkez and Robert Stojnic},
    year={2022}
}
```

```bibtex
@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}
```

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_GeorgiaTechResearchInstitute__galpaca-30b)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 40.99   |
| ARC (25-shot)         | 49.57          |
| HellaSwag (10-shot)   | 58.2    |
| MMLU (5-shot)         | 43.78         |
| TruthfulQA (0-shot)   | 41.16   |
| Winogrande (5-shot)   | 62.51   |
| GSM8K (5-shot)        | 2.81        |
| DROP (3-shot)         | 28.89         |