metadata

language:
  - en
datasets:
  - OpenAssistant/oasst1
  - databricks/databricks-dolly-15k
base_model: meta-llama/Llama-2-7b-hf
tags:
  - climate
co2_eq_emissions:
  emissions: 265800
  training_type: pre-training
  geographical_location: Washington, USA
  hardware_used: 8x NVIDIA H100 HBM

ClimateGPT-7B-FSG

⚠️ This is a research experiment to explore training from scratch on climate related data. If you're just interested in using the model, we recommend to use the Llama-2-based [ClimateGPT-7B](https://huggingface.co/eci-io/climategpt-7b).

ClimateGPT is a family of AI models designed to synthesize interdisciplinary research on climate change. ClimateGPT-7B-FSG (from scratch general) is a 7 billion parameter transformer decoder model that was pre-trained for 319.5B tokens and then continuously pre-training on a collection of 4.2B tokens from curated climate documents. The model is further instruction fine-tuned on a dataset of instruction-completion pairs manually collected by AppTek in cooperation with climate scientists. ClimateGPT-7B outperforms Llama-2-70B-Chat on our climate-specific benchmarks. The model is designed to be used together with retrieval augmentation to extend the knowledge, and increase the factuality of the model and with cascaded machine translation to increase the language coverage.

Model Details

Explore the model lineage here.

Trained by: AppTek
Powered by: Erasmus AI
Verified by: EQTYLab
Model type: decoder-only Transformer
Language(s) (NLP): English
License: ClimateGPT Community License
Continued pre-trained from: Llama-2-7B
Context length: 4K tokens
Input: Text-only data
Output: Model generates text only
Paper: Download
Website: eci.io

Uses

This is an experimental model and it is only intended to be used to reproduce our results and for LLM research. For any other use-case, we recommend to use ClimateGPT-7B, 13B or 70B
Despite the efforts from the development team to eliminate them, as every other chat-capable LLMs, this model may generate biased, offensive or inaccurate responses.

Downstream Use

ClimateGPT-7B-FSG is an instruction-tuned model that can be directly used for climate-specific question-answering applications. It was trained to perform well with retrieval augmentation and supports up to 5 references in context.

The model was trained using ChatML so the following format should be followed when prompting, including the <|im_start|>, <|im_end|> tags, system, user, context and assistant identifiers and [[0]], [[1]]] etc. tokens to indicate references.

"""
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>context
[[0]] "{reference1_title}", {reference1_year}
{reference1_text}
[[1]] "{reference2_title}", {reference2_year}
{reference2_text}
[...]<|im_end|>
<|im_start|>assistant
"""

Training

Details on the pre-training data are given in our paper.
For continued pre-training, 4.2B climate-specific tokens (tokenized by the Llama tokenizer) are used.
For instruction fine-tuning, about 272K instruction-completion pairs (both in the climate domain but also general domain) are used.

Evaluation

Detailed evaluation results are presented in our paper on our model card website: eci.io/model-card

Environmental Impact

Hardware Type: 8x NVIDIA H100 HBM
Power Consumption per GPU: 775W
Hours used: 14,288 hrs
Cloud Provider: MLFoundry
Compute Region: Washington, USA
Energy Mix: 100% Hydro Power (24g CO2eq/kWh according to IPCC 2014)
Carbon Emitted: 265.8kg CO2eq

Citation

If you find ClimateGPT is useful in your work, please cite it with:

@misc{thulke2024climategpt,
      title={ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change}, 
      author={David Thulke and Yingbo Gao and Petrus Pelser and Rein Brune and Rricha Jalota and Floris Fok and Michael Ramos and Ian van Wyk and Abdallah Nasir and Hayden Goldstein and Taylor Tragemann and Katie Nguyen and Ariana Fowler and Andrew Stanco and Jon Gabriel and Jordan Taylor and Dean Moro and Evgenii Tsymbalov and Juliette de Waal and Evgeny Matusov and Mudar Yaghi and Mohammad Shihadah and Hermann Ney and Christian Dugast and Jonathan Dotan and Daniel Erasmus},
      year={2024},
}