metadata

license: apache-2.0

Model Card for Model ID

BLING-1.4b-0.1 is part of the BLING ("Best Little Instruction-following No-GPU-required") model series.

BLING models are fine-tuned with distilled high-quality custom instruct datasets, targeted at a specific subset of instruct tasks with the objective of providing a high-quality Instruct model that is 'inference-ready' on a CPU laptop even without using any advanced quantization optimizations.

Benchmark Tests

Evaluated against the benchmark test: RAG-Instruct-Benchmark-Tester
Average of 2 Test Runs with 1 point for correct answer, 0.5 point for partial correct or blank / NF, 0.0 points for incorrect, and -1 points for hallucinations.

--Accuracy Score: 82.25 correct out of 100
--Not Found Classification: 40.0%
--Boolean: 61.25%
--Math/Logic: 8.75%
--Complex Questions (1-5): 1 (Low)
--Summarization Quality (1-5): 2 (Coherent, extractive)
--Hallucinations: No hallucinations observed in test runs.

For test run results (and good indicator of target use cases), please see the files ("core_rag_test" and "answer_sheet" in this repo).

--As a reference point, this model shows substantial improvements in results, compared with the BLING 1.0B Pythia, with fine-tuning and the base training substantially the same. The model's ability to follow instructions and answer detailed questions improves dramatically from 1.0B -> 1.4B parameters.

Model Description

Developed by: llmware
Model type: GPTNeoX instruct-trained decoder
Language(s) (NLP): English
License: Apache 2.0
Finetuned from model [optional]: EleutherAI/Pythia-1.4b-v0

Uses

The intended use of BLING models is two-fold:

Provide high-quality Instruct models that can run on a laptop for local testing. We have found it extremely useful when building a proof-of-concept, or working with sensitive enterprise data that must be closely guarded, especially in RAG use cases.
Push the state of the art for smaller Instruct-following models in the sub-7B parameter range, especially 1B-3B, as single-purpose automation tools for specific tasks through targeted fine-tuning datasets and focused "instruction" tasks.

Direct Use

BLING is designed for enterprise automation use cases, especially in knowledge-intensive industries, such as financial services, legal and regulatory industries with complex information sources. Rather than try to be "all things to all people," BLING models try to focus on a narrower set of Instructions more suitable to a ~1B parameter GPT model.

BLING is ideal for rapid prototyping, testing, and the ability to perform an end-to-end workflow locally on a laptop without having to send sensitive information over an Internet-based API.

The first BLING models have been trained for common RAG scenarios, specifically: question-answering, key-value extraction, and basic summarization as the core instruction types without the need for a lot of complex instruction verbiage - provide a text passage context, ask questions, and get clear fact-based responses.

Bias, Risks, and Limitations

Any model can provide inaccurate or incomplete information, and should be used in conjunction with appropriate safeguards and fact-checking mechanisms.

Please refer to the benchmark score and testing results for indicator as to the applicability of this model to your intended use case.
We have found that this model is reasonably effective and accurate for fact-based, extractive tasks, including key-value, question-answering, and basic summarization.

How to Get Started with the Model

The fastest way to get started with BLING is through direct import in transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM  
tokenizer = AutoTokenizer.from_pretrained("llmware/bling-1.4b-0.1")  
model = AutoModelForCausalLM.from_pretrained("llmware/bling-1.4b-0.1")

Please refer to the generation_test .py files in the Files repository, which includes 200 samples and script to test the model. The generation_test_llmware_script.py includes built-in llmware capabilities for fact-checking, as well as easy integration with document parsing and actual retrieval to swap out the test set for RAG workflow consisting of business documents.

The BLING model was fine-tuned with a simple "<human> and <bot> wrapper", so to get the best results, wrap inference entries as:

full_prompt = "\<human>\: " + my_prompt + "\n" + "\<bot>\:"

The BLING model was fine-tuned with closed-context samples, which assume generally that the prompt consists of two sub-parts:

Text Passage Context, and
Specific question or instruction based on the text passage

To get the best results, package "my_prompt" as follows:

my_prompt = {{text_passage}} + "\n" + {{question/instruction}}

If you are using a HuggingFace generation script:

# prepare prompt packaging used in fine-tuning process
new_prompt = "<human>: " + entries["context"] + "\n" + entries["query"] + "\n" + "<bot>:"

inputs = tokenizer(new_prompt, return_tensors="pt")  
start_of_output = len(inputs.input_ids[0])

#   temperature: set at 0.3 for consistency of output
#   max_new_tokens:  set at 100 - may prematurely stop a few of the summaries

outputs = model.generate(
        inputs.input_ids.to(device),
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.eos_token_id,
        do_sample=True,
        temperature=0.3,
        max_new_tokens=100,
        )

output_only = tokenizer.decode(outputs[0][start_of_output:],skip_special_tokens=True)

Citation [optional]

BLING models are built on top of EleutherAI/Pythia base - please see citation for Pythia below:

@misc{biderman2023pythia, title={Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling}, author={Stella Biderman and Hailey Schoelkopf and Quentin Anthony and Herbie Bradley and Kyle O'Brien and Eric Hallahan and Mohammad Aflah Khan and Shivanshu Purohit and USVSN Sai Prashanth and Edward Raff and Aviya Skowron and Lintang Sutawika and Oskar van der Wal}, year={2023}, eprint={2304.01373}, archivePrefix={arXiv}, primaryClass={cs.CL} }

Model Card Contact

Darren Oberst & llmware team