starcoder-toolbench / README.md
bol20162021's picture
readme
01666d6
|
raw
history blame
3.57 kB
metadata
license: bsd-3-clause

starcoder-toolbench

starcoder-toolbench is a 15 billion parameter model used for api based action generation. It is instruction tuned from starcoder on api based action generation datasets.

Model Details

Model Description

Basic Information

  • Paper: [Link]
  • Github: [Link]

Licensing

TBD

Uses

Click to expand

Direct Use

This model is intended for commercial and research use.

Out-of-Scope Use

starcoder-toolbench should NOT be used for purpose other than API based action generation.

Recommendations

Users should be made aware of the risks, biases, limitations, and restrictions of the model, which are listed down at the bottom of the page.


How to Get Started with the Model

Click to expand

Loading in model with Huggingface

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sambanovasystems/starcoder-toolbench")
model = AutoModelForCausalLM.from_pretrained("sambanovasystems/starcoder-toolbench", device_map="auto", torch_dtype="auto")

Training Details

Click to expand

Training Data

Training Procedure

We trained starcoder-toolbench on 4 80GB A100 gpu's. We started from starcoder. We finetuned it on XXX dataset. All of the code used to prepare the datasets and the scripts to run training and inference are open-sourced and freely available at [githublink here](dummy link)

Prompting Style Used For Training


Hyperparameters

  • Hardware: A100 GPU
  • Optimizer: AdamW
  • Grad accumulation: 1
  • Epochs: 8
  • Global Batch size: 16
  • Batch tokens: 16 * 2048 = 32,768 tokens
  • Learning Rate: 1e-5
  • Learning Rate Scheduler: Fixed LR
  • Weight decay: 0.1

Acknowledgment

Cite starcoder-toolbench

@software{bloomchat,
  title = {{BLOOMChat: a New Open Multilingual Chat LLM}},
  author = {SambaNova Systems, Together Computer},
  url = {https://huggingface.co/sambanovasystems/BLOOMChat-176B-v1}
  month = {5},
  year = {2023},
  version = {1.0},
}