tiktoken is a fast open-source tokenizer by OpenAI.

Given a text string (e.g., "tiktoken is great!") and an encoding (e.g., "cl100k_base"), a tokenizer can split the text string into a list of tokens (e.g., ["t", "ik", "token", " is", " great", "!"]).

Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you (a) whether the string is too long for a text model to process and (b) how much an OpenAI API call costs (as usage is priced by token).


Encodings
Encodings specify how text is converted into tokens. Different models use different encodings.

tiktoken supports three encodings used by OpenAI models:

Encoding name	OpenAI models
cl100k_base	gpt-4, gpt-3.5-turbo, text-embedding-ada-002
p50k_base	Codex models, text-davinci-002, text-davinci-003
r50k_base (or gpt2)	GPT-3 models like davinci
You can retrieve the encoding for a model using tiktoken.encoding_for_model() as follows:


0. Install tiktoken
If needed, install tiktoken with pip:

%pip install --upgrade tiktoken

1. Import tiktoken
import tiktoken

2. Load an encoding
Use tiktoken.get_encoding() to load an encoding by name.

The first time this runs, it will require an internet connection to download. Later runs won't need an internet connection.


3. Turn text into tokens with encoding.encode()
The .encode() method converts a text string into a list of token integers.

encoding.encode("tiktoken is great!")

[83, 1609, 5963, 374, 2294, 0]
Count tokens by counting the length of the list returned by .encode().


4. Turn tokens into text with encoding.decode()
.decode() converts a list of token integers to a string.

encoding.decode([83, 1609, 5963, 374, 2294, 0])

'tiktoken is great!'
Warning: although .decode() can be applied to single tokens, beware that it can be lossy for tokens that aren't on utf-8 boundaries.

For single tokens, .decode_single_token_bytes() safely converts a single integer token to the bytes it represents.


5. Comparing encodings
Different encodings vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings.


6. Counting tokens for chat completions API calls
ChatGPT models like gpt-3.5-turbo and gpt-4 use tokens in the same way as older completions models, but because of their message-based formatting, it's more difficult to count how many tokens will be used by a conversation.


Below is an example function for counting tokens for messages passed to gpt-3.5-turbo or gpt-4.

Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.

In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below:


def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")
    if model in {
        "gpt-3.5-turbo-0613",
        "gpt-3.5-turbo-16k-0613",
        "gpt-4-0314",
        "gpt-4-32k-0314",
        "gpt-4-0613",
        "gpt-4-32k-0613",
        }:
        tokens_per_message = 3
        tokens_per_name = 1
    elif model == "gpt-3.5-turbo-0301":
        tokens_per_message = 4  # every message follows {role/name}\n{content}\n
        tokens_per_name = -1  # if there's a name, the role is omitted
    elif "gpt-3.5-turbo" in model:
        print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613")
    elif "gpt-4" in model:
        print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
        return num_tokens_from_messages(messages, model="gpt-4-0613")
    else:
        raise NotImplementedError(
            f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens."""
        )
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with assistant
    return num_tokens

# let's verify the function above matches the OpenAI API response

import openai

example_messages = [
    {
        "role": "system",
        "content": "You are a helpful, pattern-following assistant that translates corporate jargon into plain English.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "New synergies will help drive top-line growth.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Things working well together will increase revenue.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Let's talk later when we're less busy about how to do better.",
    },
    {
        "role": "user",
        "content": "This late pivot means we don't have time to boil the ocean for the client deliverable.",
    },
]

for model in [
    "gpt-3.5-turbo-0301",
    "gpt-3.5-turbo-0613",
    "gpt-3.5-turbo",
    "gpt-4-0314",
    "gpt-4-0613",
    "gpt-4",
    ]:
    print(model)
    # example token count from the function defined above
    print(f"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().")
    # example token count from the OpenAI API
    response = openai.ChatCompletion.create(
        model=model,
        messages=example_messages,
        temperature=0,
        max_tokens=1,  # we're only counting input tokens here, so let's not waste tokens on the output
    )
    print(f'{response["usage"]["prompt_tokens"]} prompt tokens counted by the OpenAI API.')
    print()


How to call functions with chat models - Part 1

This notebook covers how to use the Chat Completions API in combination with external functions to extend the capabilities of GPT models. functions is an optional parameter in the Chat Completion API which can be used to provide function specifications. The purpose of this is to enable models to generate function arguments which adhere to the provided specifications. Note that the API will not actually execute any function calls. It is up to developers to execute function calls using model outputs.


How to call functions with chat models - Part 2

If the functions parameter is provided then by default the model will decide when it is appropriate to use one of the functions. The API can be forced to use a specific function by setting the function_call parameter to {'name': '<insert-function-name>'}. The API can also be forced to not use any function by setting the function_call parameter to 'none'. If a function is used, the output will contain 'finish_reason': 'function_call' in the response, as well as a function_call object that has the name of the function and the generated function arguments.


How to call functions with chat models - Overview

Overview This notebook contains the following 2 sections: How to generate function arguments: Specify a set of functions and use the API to generate function arguments. How to call functions with model generated arguments: Close the loop by actually executing functions with model generated arguments.


How to call functions with chat models - Installation

!pip install scipy !pip install tenacity !pip install tiktoken !pip install termcolor !pip install openai !pip install requests import json import openai import requests from tenacity import retry, wait_random_exponential, stop_after_attempt from termcolor import colored GPT_MODEL = 'gpt-3.5-turbo-0613'


How to call functions with chat models - Utilities

Utilities First let's define a few utilities for making calls to the Chat Completions API and for maintaining and keeping track of the conversation state. @retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3)) def chat_completion_request(messages, functions=None, function_call=None, model=GPT_MODEL): headers = { 'Content-Type': 'application/json', 'Authorization': 'Bearer ' + openai.api_key, } json_data = {'model': model, 'messages': messages} if functions is not None: json_data.update({'functions': functions}) if function_call is not None: json_data.update({'function_call': function_call}) try: response = requests.post( 'https://api.openai.com/v1/chat/completions', headers=headers, json=json_data, ) return response except Exception as e: print('Unable to generate ChatCompletion response') print(f'Exception: {e}') return e


How to call functions with chat models - Pretty Print

def pretty_print_conversation(messages): role_to_color = { 'system': 'red', 'user': 'green', 'assistant': 'blue', 'function': 'magenta', } for message in messages: if message['role'] == 'system': print(colored(f'system: {message['content']}
', role_to_color[message['role']])) elif message['role'] == 'user': print(colored(f'user: {message['content']}
', role_to_color[message['role']])) elif message['role'] == 'assistant' and message.get('function_call'): print(colored(f'assistant: {message['function_call']}
', role_to_color[message['role']])) elif message['role'] == 'assistant' and not message.get('function_call'): print(colored(f'assistant: {message['content']}
', role_to_color[message['role']])) elif message['role'] == 'function': print(colored(f'function ({message['name']}): {message['content']}
', role_to_color[message['role']]))


How to call functions with chat models - Basic Concepts

Basic concepts Let's create some function specifications to interface with a hypothetical weather API. We'll pass these function specification to the Chat Completions API in order to generate function arguments that adhere to the specification. functions = [ { 'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': { 'type': 'object', 'properties': { 'location': { 'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA', }, 'format': { 'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.', }, }, 'required': ['location', 'format'], }, }, { 'name': 'get_n_day_weather_forecast', 'description': 'Get an N-day weather forecast', 'parameters': { 'type': 'object', 'properties': { 'location': { 'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA', }, 'format': { 'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.', }, 'num_days': { 'type': 'integer', 'description': 'The number of days to forecast', } }, 'required': ['location', 'format', 'num_days'] }, }, ]


How to call functions with chat models - Weather Query

If we prompt the model about the current weather, it will respond with some clarifying questions. messages = [] messages.append({'role': 'system', 'content': 'Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.'}) messages.append({'role': 'user', 'content': 'What's the weather like today'}) chat_response = chat_completion_request( messages, functions=functions ) assistant_message = chat_response.json()['choices'][0]['message'] messages.append(assistant_message) assistant_message


How to call functions with chat models - Weather Query Response

{'role': 'assistant', 'content': 'In which city and state would you like to know the current weather?'} Once we provide the missing information, it will generate the appropriate function arguments for us. messages.append({'role': 'user', 'content': 'I'm in Glasgow, Scotland.'}) chat_response = chat_completion_request( messages, functions=functions ) assistant_message = chat_response.json()['choices'][0]['message'] messages.append(assistant_message) assistant_message


How to call functions with chat models - Weather Query Different Prompt

If we prompt it differently, we can get it to target the other function we've told it about. messages = [] messages.append({'role': 'system', 'content': 'Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.'}) messages.append({'role': 'user', 'content': 'what is the weather going to be like in Glasgow, Scotland over the next x days'}) chat_response = chat_completion_request( messages, functions=functions ) assistant_message = chat_response.json()['choices'][0]['message'] messages.append(assistant_message) assistant_message


How to call functions with chat models - Weather Query Response 2

{'role': 'assistant', 'content': 'Sure, I can help you with that. Please provide me with the number of days you want to forecast for.'} Once again, the model is asking us for clarification because it doesn't have enough information yet. In this case it already knows the location for the forecast, but it needs to know how many days are required in the forecast. messages.append({'role': 'user', 'content': '5 days'}) chat_response = chat_completion_request( messages, functions=functions ) chat_response.json()['choices'][0]


How to call functions with chat models - Force Specific Function

Forcing the use of specific functions or no function We can force the model to use a specific function, for example get_n_day_weather_forecast by using the function_call argument. By doing so, we force the model to make assumptions about how to use it. # in this cell we force the model to use get_n_day_weather_forecast messages = [] messages.append({'role': 'system', 'content': 'Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.'}) messages.append({'role': 'user', 'content': 'Give me a weather report for Toronto, Canada.'}) chat_response = chat_completion_request( messages, functions=functions, function_call={'name': 'get_n_day_weather_forecast'} ) chat_response.json()['choices'][0]['message']


How to call functions with chat models - Without Forcing Function

# if we don't force the model to use get_n_day_weather_forecast it may not messages = [] messages.append({'role': 'system', 'content': 'Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.'}) messages.append({'role': 'user', 'content': 'Give me a weather report for Toronto, Canada.'}) chat_response = chat_completion_request( messages, functions=functions ) chat_response.json()['choices'][0]['message']


How to call functions with chat models - Force No Function

We can also force the model to not use a function at all. By doing so we prevent it from producing a proper function call. messages = [] messages.append({'role': 'system', 'content': 'Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.'}) messages.append({'role': 'user', 'content': 'Give me the current weather (use Celcius) for Toronto, Canada.'}) chat_response = chat_completion_request( messages, functions=functions, function_call='none' ) chat_response.json()['choices'][0]['message']


How to call functions with chat models - SQL Functions

How to call functions with model generated arguments In our next example, we'll demonstrate how to execute functions whose inputs are model-generated, and use this to implement an agent that can answer questions for us about a database. For simplicity we'll use the Chinook sample database. Note: SQL generation can be high-risk in a production environment since models are not perfectly reliable at generating correct SQL.


How to call functions with chat models - SQL Function Specification

Specifying a function to execute SQL queries First let's define some helpful utility functions to extract data from a SQLite database. import sqlite3 conn = sqlite3.connect('data/Chinook.db') print('Opened database successfully')


How to call functions with chat models - Database Schema Functions

def get_table_names(conn): 'Return a list of table names' table_names = [] tables = conn.execute('SELECT name FROM sqlite_master WHERE type='table';') for table in tables.fetchall(): table_names.append(table[0]) return table_names def get_column_names(conn, table_name): 'Return a list of column names.' column_names = [] columns = conn.execute(f'PRAGMA table_info({table_name});').fetchall() for col in columns: column_names.append(col[1]) return column_names def get_database_info(conn): 'Return a list of dicts containing the table name and columns for each table in the database' table_dicts = [] for table_name in get_table_names(conn): columns_names = get_column_names(conn, table_name) table_dicts.append({'table_name': table_name, 'column_names': columns_names}) return table_dicts


How to call functions with chat models - Database Schema Representation

database_schema_dict = get_database_info(conn) database_schema_string = '\n'.join( [ f'Table: {table['table_name']}'\nColumns: {', '.join(table['column_names'])}' for table in database_schema_dict ] )


How to call functions with chat models - SQL Function Specification 2

As before, we'll define a function specification for the function we'd like the API to generate arguments for. Notice that we are inserting the database schema into the function specification. This will be important for the model to know about. functions = [ { 'name': 'ask_database', 'description': 'Use this function to answer user questions about music. Input should be a fully formed SQL query.', 'parameters': { 'type': 'object', 'properties': { 'query': { 'type': 'string', 'description': f' SQL query extracting info to answer the user's question. SQL should be written using this database schema: {database_schema_string} The query should be returned in plain text, not in JSON.', }, }, 'required': ['query'], }, }, ]


How to call functions with chat models - Execute SQL Queries

Executing SQL queries Now let's implement the function that will actually excute queries against the database. def ask_database(conn, query): 'Function to query SQLite database with a provided SQL query.' try: results = str(conn.execute(query).fetchall()) except Exception as e: results = f'query failed with error: {e}' return results def execute_function_call(message): if message['function_call']['name'] == 'ask_database': query = json.loads(message['function_call']['arguments'])['query'] results = ask_database(conn, query) else: results = f'Error: function {message['function_call']['name']} does not exist' return results


How to call functions with chat models - SQL Query Example

messages = [] messages.append({'role': 'system', 'content': 'Answer user questions by generating SQL queries against the Chinook Music Database.'}) messages.append({'role': 'user', 'content': 'Hi, who are the top 5 artists by number of tracks?'}) chat_response = chat_completion_request(messages, functions) assistant_message = chat_response.json()['choices'][0]['message'] messages.append(assistant_message) if assistant_message.get('function_call'): results = execute_function_call(assistant_message) messages.append({'role': 'function', 'name': assistant_message['function_call']['name'], 'content': results}) pretty_print_conversation(messages)


How to call functions with chat models - SQL Query Example 2

messages.append({'role': 'user', 'content': 'What is the name of the album with the most tracks?'}) chat_response = chat_completion_request(messages, functions) assistant_message = chat_response.json()['choices'][0]['message'] messages.append(assistant_message) if assistant_message.get('function_call'): results = execute_function_call(assistant_message) messages.append({'role': 'function', 'content': results, 'name': assistant_message['function_call']['name']}) pretty_print_conversation(messages)


Data preparation and analysis for chat model fine-tuning

This notebook serves as a tool to preprocess and analyze the chat dataset used for fine-tuning a chat model. It checks for format errors, provides basic statistics, and estimates token counts for fine-tuning costs. The method shown here corresponds to legacy fine-tuning for models like babbage-002 and davinci-002. For fine-tuning gpt-3.5-turbo, see the current fine-tuning page.


We first load the chat dataset from an example JSONL file.

data_path = "data/toy_chat_fine_tuning.jsonl"

# Load the dataset
with open(data_path, 'r', encoding='utf-8') as f:
    dataset = [json.loads(line) for line in f]

# Initial dataset stats
print("Num examples:", len(dataset))
print("First example:")
for message in dataset[0]["messages"]:
    print(message)


We can perform a variety of error checks to validate that each conversation in the dataset adheres to the format expected by the fine-tuning API. Errors are categorized based on their nature for easier debugging.

Data Type Check: Checks whether each entry in the dataset is a dictionary (dict). Error type: data_type.
Presence of Message List: Checks if a messages list is present in each entry. Error type: missing_messages_list.
Message Keys Check: Validates that each message in the messages list contains the keys role and content. Error type: message_missing_key.
Unrecognized Keys in Messages: Logs if a message has keys other than role, content, and name. Error type: message_unrecognized_key.
Role Validation: Ensures the role is one of "system", "user", or "assistant". Error type: unrecognized_role.
Content Validation: Verifies that content has textual data and is a string. Error type: missing_content.
Assistant Message Presence: Checks that each conversation has at least one message from the assistant. Error type: example_missing_assistant_message.
The code below performs these checks, and outputs counts for each type of error found are printed. This is useful for debugging and ensuring the dataset is ready for the next steps.


Lets define a few helpful utilities to be used in the rest of the notebook.

encoding = tiktoken.get_encoding("cl100k_base")

# not exact!
# simplified from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3
    return num_tokens


def num_assistant_tokens_from_messages(messages):
    num_tokens = 0
    for message in messages:
        if message["role"] == "assistant":
            num_tokens += len(encoding.encode(message["content"]))
    return num_tokens


def print_distribution(values, name):
    print(f"\n#### Distribution of {name}:")
    print(f"min / max: {min(values)}, {max(values)}")
    print(f"mean / median: {np.mean(values)}, {np.median(values)}")
    print(f"p5 / p95: {np.quantile(values, 0.1)}, {np.quantile(values, 0.9)}")


With some lightweight analysis we can identify potential issues in the dataset, like missing messages, and provide statistical insights into message and token counts.

Missing System/User Messages: Counts the number of conversations missing a "system" or "user" message. Such messages are critical for defining the assistant's behavior and initiating the conversation.
Number of Messages Per Example: Summarizes the distribution of the number of messages in each conversation, providing insight into dialogue complexity.
Total Tokens Per Example: Calculates and summarizes the distribution of the total number of tokens in each conversation. Important for understanding fine-tuning costs.
Tokens in Assistant's Messages: Calculates the number of tokens in the assistant's messages per conversation and summarizes this distribution. Useful for understanding the assistant's verbosity.
Token Limit Warnings: Checks if any examples exceed the maximum token limit (4096 tokens), as such examples will be truncated during fine-tuning, potentially resulting in data loss.


# Warnings and tokens counts
n_missing_system = 0
n_missing_user = 0
n_messages = []
convo_lens = []
assistant_message_lens = []

for ex in dataset:
    messages = ex["messages"]
    if not any(message["role"] == "system" for message in messages):
        n_missing_system += 1
    if not any(message["role"] == "user" for message in messages):
        n_missing_user += 1
    n_messages.append(len(messages))
    convo_lens.append(num_tokens_from_messages(messages))
    assistant_message_lens.append(num_assistant_tokens_from_messages(messages))

print("Num examples missing system message:", n_missing_system)
print("Num examples missing user message:", n_missing_user)
print_distribution(n_messages, "num_messages_per_example")
print_distribution(convo_lens, "num_total_tokens_per_example")
print_distribution(assistant_message_lens, "num_assistant_tokens_per_example")
n_too_long = sum(l > 4096 for l in convo_lens)
print(f"\n{n_too_long} examples may be over the 4096 token limit, they will be truncated during fine-tuning")


In this final section, we estimate the total number of tokens that will be used for fine-tuning, which allows us to approximate the cost. It is worth noting that the duration of the fine-tuning jobs will also increase with the token count.

# Pricing and default n_epochs estimate
MAX_TOKENS_PER_EXAMPLE = 4096

TARGET_EPOCHS = 3
MIN_TARGET_EXAMPLES = 100
MAX_TARGET_EXAMPLES = 25000
MIN_DEFAULT_EPOCHS = 1
MAX_DEFAULT_EPOCHS = 25

n_epochs = TARGET_EPOCHS
n_train_examples = len(dataset)
if n_train_examples * TARGET_EPOCHS < MIN_TARGET_EXAMPLES:
    n_epochs = min(MAX_DEFAULT_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples)
elif n_train_examples * TARGET_EPOCHS > MAX_TARGET_EXAMPLES:
    n_epochs = max(MIN_DEFAULT_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples)

n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens)
print(f"Dataset has ~{n_billing_tokens_in_dataset} tokens that will be charged for during training")
print(f"By default, you'll train for {n_epochs} epochs on this dataset")
print(f"By default, you'll be charged for ~{n_epochs * n_billing_tokens_in_dataset} tokens")

Dataset has ~4306 tokens that will be charged for during training
By default, you'll train for 20 epochs on this dataset
By default, you'll be charged for ~86120 tokens
See https://openai.com/pricing to estimate total costs.


Wikipedia article section:
Curling at the 2022 Winter Olympics

==Results summary==

===Women's tournament===

====Playoffs====

=====Gold medal game=====
''Sunday, 20 February, 9:05''
{{#lst:Curling at the 2022 Winter Olympics – Women's tournament|GM}}
{{Player percentages
| team1 = {{flagIOC|JPN|2022 Winter}}
| [[Yurika Yoshida]] | 97%
| [[Yumi Suzuki]] | 82%
| [[Chinami Yoshida]] | 64%
| [[Satsuki Fujisawa]] | 69%
| teampct1 = 78%
| team2 = {{flagIOC|GBR|2022 Winter}}
| [[Hailey Duff]] | 90%
| [[Jennifer Dodds]] | 89%
| [[Vicky Wright]] | 89%
| [[Eve Muirhead]] | 88%
| teampct2 = 89%
}}


Curling at the 2022 Winter Olympics - Medal summary

Wikipedia article section:
Curling at the 2022 Winter Olympics

==Medal summary==

===Medal table===
{{Medals table
| caption =
| host =
| flag_template = flagIOC
| event = 2022 Winter
| team =
| gold_CAN = 0 | silver_CAN = 0 | bronze_CAN = 1
| gold_ITA = 1 | silver_ITA = 0 | bronze_ITA = 0
| gold_NOR = 0 | silver_NOR = 1 | bronze_NOR = 0
| gold_SWE = 1 | silver_SWE = 0 | bronze_SWE = 2
| gold_GBR = 1 | silver_GBR = 1 | bronze_GBR = 0
| gold_JPN = 0 | silver_JPN = 1 | bronze_JPN - 0
}}


Curling at the 2022 Winter Olympics - Men's tournament

Wikipedia article section:
Curling at the 2022 Winter Olympics

==Results summary==

===Men's tournament===

====Playoffs====

=====Gold medal game=====
''Saturday, 19 February, 14:50''
{{#lst:Curling at the 2022 Winter Olympics – Men's tournament|GM}}
{{Player percentages
| team1 = {{flagIOC|GBR|2022 Winter}}
| [[Hammy McMillan Jr.]] | 95%
| [[Bobby Lammie]] | 80%
| [[Grant Hardie]] | 94%
| [[Bruce Mouat]] | 89%
| teampct1 = 90%
| team2 = {{flagIOC|SWE|2022 Winter}}
| [[Christoffer Sundgren]] | 99%
| [[Rasmus Wranå]] | 95%
| [[Oskar Eriksson]] | 93%
| [[Niklas Edin]] | 87%
| teampct2 = 94%
}}


Curling at the 2022 Winter Olympics - Medalists

Wikipedia article section:
Curling at the 2022 Winter Olympics

==Medal summary==

===Medalists===
{| {{MedalistTable|type=Event|columns=1}}
|-
|Men
{{DetailsLink|Curling at the 2022 Winter Olympics – Men's tournament}}
|{{flagIOC|SWE|2022 Winter}}
[[Niklas Edin]]
[[Oskar Eriksson]]
[[Rasmus Wranå]]
[[Christoffer Sundgren]]
[[Daniel Magnusson (curler)|Daniel Magnusson]]
|{{flagIOC|GBR|2022 Winter}}
[[Bruce Mouat]]
[[Grant Hardie]]
[[Bobby Lammie]]
[[Hammy McMillan Jr.]]
[[Ross Whyte]]
|{{flagIOC|CAN|2022 Winter}}
[[Brad Gushue]]
[[Mark Nichols (curler)|Mark Nichols]]
[[Brett Gallant]]
[[Geoff Walker (curler)|Geoff Walker]]
[[Marc Kennedy]]
|-
|Women
{{DetailsLink|Curling at the 2022 Winter Olympics – Women's tournament}}
|{{flagIOC|GBR|2022 Winter}}
[[Eve Muirhead]]
[[Vicky Wright]]
[[Jennifer Dodds]]
[[Hailey Duff]]
[[Mili Smith]]
|{{flagIOC|JPN|2022 Winter}}
[[Satsuki Fujisawa]]
[[Chinami Yoshida]]
[[Yumi Suzuki]]
[[Yurika Yoshida]]
[[Kotomi Ishizaki]]
|{{flagIOC|SWE|2022 Winter}}
[[Anna Hasselborg]]
[[Sara McManus]]
[[Agnes Knochenhauer]]
[[Sofia Mabergs]]
[[Johanna Heldin]]
|-
|Mixed doubles
{{DetailsLink|Curling at the 2022 Winter Olympics – Mixed doubles tournament}}
|{{flagIOC|ITA|2022 Winter}}
[[Stefania Constantini]]
[[Amos Mosaner]]
|{{flagIOC|NOR|2022 Winter}}
[[Kristin Skaslien]]
[[Magnus Nedregotten]]
|{{flagIOC|SWE|2022 Winter}}
[[Almida de Val]]
[[Oskar Eriksson]]
|}


Curling at the 2022 Winter Olympics - Men's tournament (Bronze medal game)

Wikipedia article section:
Curling at the 2022 Winter Olympics

==Results summary==

===Men's tournament===

====Playoffs====

=====Bronze medal game=====
''Friday, 18 February, 14:05''
{{#lst:Curling at the 2022 Winter Olympics – Men's tournament|BM}}
{{Player percentages
| team1 = {{flagIOC|USA|2022 Winter}}
| [[John Landsteiner]] | 80%
| [[Matt Hamilton (curler)|Matt Hamilton]] | 86%
| [[Chris Plys]] | 74%
| [[John Shuster]] | 69%
| teampct1 = 77%
| team2 = {{flagIOC|CAN|2022 Winter}}
| [[Geoff Walker (curler)|Geoff Walker]] | 84%
| [[Brett Gallant]] | 86%
| [[Mark Nichols (curler)|Mark Nichols]] | 78%
| [[Brad Gushue]] | 78%
| teampct2 = 82%
}}


Curling at the 2022 Winter Olympics - Teams (Mixed doubles)

Wikipedia article section:
Curling at the 2022 Winter Olympics

==Teams==

===Mixed doubles===
{| class=wikitable
|-
!width=200|{{flagIOC|AUS|2022 Winter}}
!width=200|{{flagIOC|CAN|2022 Winter}}
!width=200|{{flagIOC|CHN|2022 Winter}}
!width=200|{{flagIOC|CZE|2022 Winter}}
!width=200|{{flagIOC|GBR|2022 Winter}}
|-
|
'''Female:''' [[Tahli Gill]]

'''Male:''' [[Dean Hewitt]]
|
'''Female:''' [[Rachel Koman
Male: John Morris]]
|
'''Female:''' Fan S
Male: L Z
|
'''Female:''' Z Paulov
Male: T Paul
|
'''Female:''' Jennifer Dodds
Male: B Mouat
|-
!width=200|{{flagIOC|ITA|2022 Winter}}
!width=200|{{flagIOC|NOR|2022 Winter}}
!width=200|{{flagIOC|SWE|2022 Winter}}
!width=200|{{flagIOC|SUI|2022 Winter}}
!width=200|{{flagIOC|USA|2022 Winter}}
|-
|
'''Female:''' Stefania Constantini
'''Male:''' Amos Mosaner
|
'''Female:''' Kristin S
Male: Magnus Nedregotten
|
'''Female:''' A de Val
Male: O Eriksson
|
'''Female:''' Jenny Perret
Male: Martin Rios
|
'''Female:''' Vicky Persinger
Male: Chris Plys
|}


Curling at the 2022 Winter Olympics - Women's tournament (Bronze medal game)

Wikipedia article section:
Curling at the 2022 Winter Olympics

==Results summary==

===Women's tournament===

====Playoffs====

=====Bronze medal game=====
''Saturday, 19 February, 20:05''
{{#lst:Curling at the 2022 Winter Olympics – Women's tournament|BM}}
{{Player percentages
| team1 = {{flagIOC|SUI|2022 Winter}}
| [[Melanie Barbezat]] | 79%
| [[Esther Neuenschwander]] | 75%
| [[Silvana Tirinzoni]] | 81%
| [[Alina Pätz]] | 64%
| teampct1 = 75%
| team2 = {{flagIOC|SWE|2022 Winter}}
| [[Sofia Mabergs]] | 89%
| [[Agnes Knochenhauer]] | 80%
| [[Sara McManus]] | 81%
| [[Anna Hasselborg]] | 76%
| teampct2 = 82%
}}


Curling at the 2022 Winter Olympics - Mixed doubles tournament (Gold medal game)

Wikipedia article section:
Curling at the 2022 Winter Olympics

==Results summary==

===Mixed doubles tournament===

====Playoffs====

=====Gold medal game=====
''Tuesday, 8 February, 20:05''
{{#lst:Curling at the 2022 Winter Olympics – Mixed doubles tournament|GM}}
{| class="wikitable"
!colspan=4 width=400|Player percentages
|-
!colspan=2 width=200 style="white-space:nowrap;"| {{flagIOC|ITA|2022 Winter}}
!colspan=2 width=200 style="white-space:nowrap;"| {{flagIOC|NOR|2022 Winter}}
|-
| [[Stefania Constantini]] || 83%
| [[Kristin Skaslien]] || 70%
|-
| [[Amos Mosaner]] || 90%
| [[Magnus Nedregotten]] || 69%
|-
| '''Total''' || 87%
| '''Total''' || 69%
|}


ChatGPT is powered by gpt-3.5-turbo and gpt-4, OpenAI's most advanced models. You can build your own applications with gpt-3.5-turbo or gpt-4 using the OpenAI API. Chat models take a series of messages as input, and return an AI-written message as output. This guide illustrates the chat format with a few example API calls.


# if needed, install and/or upgrade to the latest version of the OpenAI Python library %pip install --upgrade openai import the OpenAI Python library for calling the OpenAI API import openai


A chat API call has two required inputs: model: the name of the model you want to use (e.g., gpt-3.5-turbo, gpt-4, gpt-3.5-turbo-0613, gpt-3.5-turbo-16k-0613) messages: a list of message objects, where each object has two required fields: role: the role of the messenger (either system, user, or assistant) content: the content of the message (e.g., Write me a beautiful poem) Messages can also contain an optional name field, which gives the messenger a name. E.g., example-user, Alice, BlackbeardBot. Names may not contain spaces.


response['choices'][0]['message']['content'] 'Orange who?' Even non-conversation-based tasks can fit into the chat format, by placing the instruction in the first user message.


Best practices for instructing models may change from model version to model version. The advice that follows applies to gpt-3.5-turbo-0301 and may not apply to future models. System messages The system message can be used to prime the assistant with different personalities or behaviors.


In some cases, it's easier to show the model what you want rather than tell the model what you want. One way to show the model what you want is with faked example messages.


Transforming the few-shot example above, we could write:

The business jargon translation example, but with example names for the example messages


Not every attempt at engineering conversations will succeed at first.

If your first attempts fail, don't be afraid to experiment with different ways of priming or conditioning the model.


When you submit your request, the API transforms the messages into a sequence of tokens. The number of tokens used affects: the cost of the request the time it takes to generate the response when the reply gets cut off from hitting the maximum token limit (4,096 for gpt-3.5-turbo or 8,192 for gpt-4)


You can use the following function to count the number of tokens that a list of messages will use. Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.


```python
def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")
    if model in {
        "gpt-3.5-turbo-0613",
        "gpt-3.5-turbo-16k-0613",
        "gpt-4-0314",
        "gpt-4-32k-0314",
        "gpt-4-0613",
        "gpt-4-32k-0613",
        }:
        tokens_per_message = 3
        tokens_per_name = 1
    elif model == "gpt-3.5-turbo-0301":
        tokens_per_message = 4  # every message follows {role/name}\n{content}\n
        tokens_per_name = -1  # if there's a name, the role is omitted
    elif "gpt-3.5-turbo" in model:
        print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613")
    elif "gpt-4" in model:
        print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
        return num_tokens_from_messages(messages, model="gpt-4-0613")
    else:
        raise NotImplementedError(
            f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens."""
        )
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with assistant
    return num_tokens
```


By default, when you request a completion from the OpenAI, the entire completion is generated before being sent back in a single response.

If you're generating long completions, waiting for the response can take many seconds.

To get responses sooner, you can 'stream' the completion as it's being generated. This allows you to start printing or processing the beginning of the completion before the full completion is finished.


To stream completions, set stream=True when calling the chat completions or completions endpoints. This will return an object that streams back the response as data-only server-sent events. Extract chunks from the delta field rather than the message field.


Note that using stream=True in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate. which has implications for approved usage.

Another small drawback of streaming responses is that the response no longer includes the usage field to tell you how many tokens were consumed. After receiving and combining all of the responses, you can calculate this yourself using tiktoken.


Below, this notebook shows:

What a typical chat completion response looks like
What a streaming chat completion response looks like
How much time is saved by streaming a chat completion
How to stream non-chat completions (used by older models like text-davinci-003)


# imports
import openai  # for OpenAI API calls
import time  # for measuring time duration of API calls

1. What a typical chat completion response looks like
With a typical ChatCompletions API call, the response is first computed and then returned all at once.


The reply can be extracted with response['choices'][0]['message'].

The content of the reply can be extracted with response['choices'][0]['message']['content'].

reply = response['choices'][0]['message']
print(f"Extracted reply:
{reply}")

reply_content = response['choices'][0]['message']['content']
print(f"Extracted content:
{reply_content}")


With a streaming API call, the response is sent back incrementally in chunks via an event stream. In Python, you can iterate over these events with a for loop.


As you can see above, streaming responses have a delta field rather than a message field. delta can hold things like:

a role token (e.g., {'role': 'assistant'})
a content token (e.g., {'content': '\n\n'})
nothing (e.g., {}), when the stream is over


3. How much time is saved by streaming a chat completion
Now let's ask gpt-3.5-turbo to count to 100 again, and see how long it takes.


# Example of an OpenAI ChatCompletion request with stream=True
# https://platform.openai.com/docs/guides/chat

# record the time before the request is sent
start_time = time.time()


# send a ChatCompletion request to count to 100
response = openai.ChatCompletion.create(
    model='gpt-3.5-turbo',
    messages=[
        {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}
    ],
    temperature=0,
)


# calculate the time it took to receive the response
response_time = time.time() - start_time

# print the time delay and text received
print(f"Full response received {response_time:.2f} seconds after request")
print(f"Full response received:
{response}")


How to stream non-chat completions (used by older models like text-davinci-003)

A typical completion request
With a typical Completions API call, the text is first computed and then returned all at once.


How to stream non-chat completions (used by older models like text-davinci-003) (cont.)

# Example of an OpenAI Completion request
# https://beta.openai.com/docs/api-reference/completions/create

# record the time before the request is sent
start_time = time.time()


# send a Completion request to count to 100
response = openai.Completion.create(
    model='text-davinci-002',
    prompt='1,2,3,',
    max_tokens=193,
    temperature=0,
)


# calculate the time it took to receive the response
response_time = time.time() - start_time

# extract the text from the response
completion_text = response['choices'][0]['text']


# print the time delay and text received
print(f"Full response received {response_time:.2f} seconds after request")
print(f"Full text received: {completion_text}")


A streaming completion request
With a streaming Completions API call, the text is sent back via a series of events. In Python, you can iterate over these events with a for loop.


# Example of an OpenAI Completion request, using the stream=True option
# https://beta.openai.com/docs/api-reference/completions/create

# record the time before the request is sent
start_time = time.time()


# send a Completion request to count to 100
response = openai.Completion.create(
    model='text-davinci-002',
    prompt='1,2,3,',
    max_tokens=193,
    temperature=0,
    stream=True,  # this time, we set stream=True
)


# create variables to collect the stream of events
collected_events = []
completion_text = ''
# iterate through the stream of events
for event in response:
    event_time = time.time() - start_time  # calculate the time delay of the event
    collected_events.append(event)  # save the event response/...sponse received {event_time:.2f} seconds after request")
print(f"Full text received: {completion_text}
 However, with the streaming request, we received the first token after 0.18 seconds, and subsequent tokens every ~0.01-0.02 seconds")


When GPT-3 fails on a task, what should you do?

Search for a better prompt that elicits more reliable answers?
Invest in thousands of examples to fine-tune a custom model?
Assume the model is incapable of the task, and move on?
There is no simple answer - it depends. However, if your task involves logical reasoning or complexity, consider trying the techniques in this article to build more reliable, high-performing prompts.


If you were asked to multiply 13 by 17, would the answer pop immediately into your mind? For most of us, probably not. Yet, that doesn't mean humans are incapable of two-digit multiplication. With a few seconds, and some pen and paper, it's not too taxing to work out that 13 x 17 = 130 + 70 + 21 = 221.

Similarly, if you give GPT-3 a task that's too complex to do in the time it takes to calculate its next token, it may confabulate an incorrect guess. Yet, akin to humans, that doesn't necessarily mean the model is incapable of the task. With some time and space to reason things out, the model still may be able to answer reliably.


As an example, if you ask text-davinci-002...

As an example, if you ask text-davinci-002 the following math problem about juggling balls, it answers incorrectly:

Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there?
A:
There are 8 blue golf balls.
Does this mean that GPT-3 cannot do simple math problems? No; in fact, it turns out that by prompting the model with Let's think step by step, the model solves the problem reliably:

Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there?
A: Let's think step by step.
There are 16 balls in total.
Half of the balls are golf balls.
That means that there are 8 golf balls.
Half of the golf balls are blue.
That means that there are 4 blue golf balls.
Of course, it's hard to tell from only a single example whether this Let's think step by step trick actually works in general or just got lucky on this particular problem. But it really does work. On a benchmark of word math problems, the Let's think step by step trick raised GPT-3's solve rate massively, from a worthless 18% to a decent 79%!


When learning to work with GPT-3, one common conceptual mistake is to believe that its capabilities are fixed across all contexts. E.g., if GPT-3 gets a simple logic question wrong, then it must be incapable of simple logic.

But as the Let's think step by step example illustrates, apparent failures of GPT-3 can sometimes be remedied with a better prompt that helps the model steer itself toward the correct output.


How to improve reliability on complex tasks

The rest of this article shares techniques for improving reliability of large language models on complex tasks. Although some of the techniques are specific to certain types of problems, many of them are built upon general principles that can be applied to a wide range of tasks, e.g.:

Give clearer instructions
Split complex tasks into simpler subtasks
Structure the instruction to keep the model on task
Prompt the model to explain before answering
Ask for justifications of many possible answers, and then synthesize
Generate many outputs, and then use the model to pick the best one
Fine-tune custom models to maximize performance
Split complex tasks into simpler tasks
One way to give a model more time and space to think is to break tasks into simpler pieces.


Techniques to improve reliability - Part 1

As an example, consider a task where we ask the model a multiple-choice question about some text - in this case, a game of Clue. When asked directly, text-davinci-002 isn't able to put clues 3 & 5 together, and answers incorrectly:


Techniques to improve reliability - Part 2

Use the following clues to answer the following multiple-choice question.

Clues:
1. Miss Scarlett was the only person in the lounge.
2. The person with the pipe was in the kitchen.
3. Colonel Mustard was the only person in the observatory.
4. Professor Plum was not in the library nor the billiard room.
5. The person with the candlestick was in the observatory.


Techniques to improve reliability - Part 3

Question: Was Colonel Mustard in the observatory with the candlestick?
(a) Yes; Colonel Mustard was in the observatory with the candlestick
(b) No; Colonel Mustard was not in the observatory with the candlestick
(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick

Solution:


Techniques to improve reliability - Part 4

(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick
Although clues 3 and 5 establish that Colonel Mustard was the only person in the observatory and that the person in the observatory had the candlestick, the model fails to combine them into a correct answer of (a) Yes.


Techniques to improve reliability - Part 5

However, instead of asking for the answer directly, we can split the task into three pieces:


Techniques to improve reliability - Part 6

First, go through the clues one by one and consider whether the clue is potentially relevant
Second, combine the relevant clues to reason out the answer to the question
Third, write the final answer: either (a), (b), or (c)


Techniques to improve reliability - Part 7

Use the following clues to answer the following multiple-choice question, using the following procedure:
(1) First, go through the clues one by one and consider whether the clue is potentially relevant
(2) Second, combine the relevant clues to reason out the answer to the question
(3) Third, map the answer to one of the multiple choice answers: either (a), (b), or (c)


Techniques to improve reliability - Part 8

Clues:
1. Miss Scarlett was the only person in the lounge.
2. The person with the pipe was in the kitchen.
3. Colonel Mustard was the only person in the observatory.
4. Professor Plum was not in the library nor the billiard room.
5. The person with the candlestick was in the observatory.


Techniques to improve reliability - Part 9

Question: Was Colonel Mustard in the observatory with the candlestick?
(a) Yes; Colonel Mustard was in the observatory with the candlestick
(b) No; Colonel Mustard was not in the observatory with the candlestick
(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick


Techniques to improve reliability - Part 10

Solution:
(1) First, go through the clues one by one and consider whether the clue is potentially relevant:
Clue 1: Miss Scarlett was the only person in the lounge. This clue is not relevant.
Clue 2: The person with the pipe was in the kitchen. This clue is not relevant.
Clue 3: Colonel Mustard was the only person in the observatory. This clue is relevant.


Techniques to improve reliability - Part 11

Clue 4: Professor Plum was not in the library nor the billiard room. This clue is not relevant.
Clue 5: The person with the candlestick was in the observatory. This clue is relevant.

(2) Second, combine the relevant clues to reason out the answer to the question:
Clue 3 and Clue 5 both say that the person with the candlestick was in the observatory. Colonel Mustard was the only person in the observatory, so Colonel Mustard was the person with the candlestick.


Techniques to improve reliability - Part 12

(3) Third, write the final answer:
(a) Yes; Colonel Mustard was in the observatory with the candlestick
By giving the model more time and space to think, and guiding it along a reasoning plan, it's able to figure out the correct answer of (a) Yes.


Techniques to improve reliability - Part 13

Another benefit of splitting complex instructions into smaller subtasks is that it can help keep the model focused on each subtask.


Techniques to improve reliability - Part 14

For example, if we ask text-davinci-002 to summarize a text in its original language, the model can lapse back into English:


Techniques to improve reliability - Part 15

Summarize the text using the original language of the text. The summary should be one sentence long.


Techniques to improve reliability - Part 16

Text:
"La estadística (la forma femenina del término alemán Statistik, derivado a su vez del italiano statista, 'hombre de Estado') es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad.


Techniques to improve reliability - Part 17

Summary:
The text explains that statistics is a science that studies the variability, collection, organization, analysis, interpretation, and presentation of data, as well as the random process that generates them following the laws of probability.


Techniques to improve reliability - Part 18

However, if we first ask the model to identify the language of the text, and then summarize the text, it becomes more reliable:


Techniques to improve reliability - Part 19

First, identify the language of the text. Second, summarize the text using the original language of the text. The summary should be one sentence long.


Techniques to improve reliability - Part 20

Language:
Spanish

La estadística es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad.


Techniques to improve reliability - Part 21

Prompt the model to explain before answering
Another powerful technique for improving the reliability of answers is to prompt the model to gradually reason out the answer rather than jumping immediately to the final answer. By 'thinking aloud' the model can be far more likely to arrive at the correct answer.


Techniques to improve reliability - Part 22

Zero-shot
Method
Published by Takeshi Kojima et al. in 2022, the easiest way to prompt a model to reason out the answer is to simply prepend answers with Let's think step by step


Source: Large Language Models are Zero-Shot Reasoners by Takeshi Kojima et al. (2022).

Results
Applying this simple trick to the MultiArith math dataset, the authors found Let's think step by step quadrupled the accuracy, from 18% to 79%!

zero-shot reasoning example


Techniques to improve reliability (continued)

Implications
Although the Let's think step by step trick works well on math problems, it's not effective on all tasks. The authors found that it was most helpful for multi-step arithmetic problems, symbolic reasoning problems, strategy problems, and other reasoning problems. It didn't help with simple math problems or common sense questions, and presumably wouldn't help with many other non-reasoning tasks either.

zero-shot reasoning example


To learn more, read the full paper.

If you apply this technique to your own tasks, don't be afraid to experiment with customizing the instruction. Let's think step by step is rather generic, so you may find better performance with instructions that hew to a stricter format customized to your use case. For example, you can try more structured variants like First, think step by step about why X might be true. Second, think step by step about why Y might be true. Third, think step by step about whether X or Y makes more sense.. And you can even give the model an example format to help keep it on track, e.g.:


Question: Can I claim a federal tax credit for my Toyota Prius Prime bought in 2021?

(1) For each criterion, determine whether it is met by the vehicle purchase
- Does the vehicle have at least four wheels? Let's think step by step.
 The Toyota Prius Prime has four wheels, so the answer is yes.
- Does the vehicle weigh less than 14,000 pounds? Let's think step by step. The Toyota Prius Prime weighs less than 14,000 pounds, so the answer is yes.
- Does the vehicle draw energy from a battery with at least 4 kilowatt hours that may be recharged from an external source? Let's think step by step. The Toyota Prius Prime has a battery with at least 4 kilowatt hours that may be recharged from an external source, so the answer is yes.
- Was the vehicle purchased in a year before 2022? Let's think step by step. The Toyota Prius Prime was purchased in 2021, which is before 2022, so the answer is yes.
- Was the vehicle purchased in a year after 2022? N/A
- If so, is the vehicle present in the following list of North American-assembled vehicles? N/A

(2) After considering each criterion in turn, phrase the final answer as "Because of {reasons}, the answer is likely {yes or no}."

Because the Toyota Prius Prime meets all of the criteria for a federal tax credit, the answer is likely yes.


Method
Prompting the model to reason out its answers can be done in many ways. One way is to demonstrate with a few examples ('few-shot'), as studied by Jason Wei and Denny Zhou et al. from Google. Here's an example few-shot chain-of-thought prompt:


Results
Testing on grade school math problems, the authors found that chain of thought prompting tripled the solve rate, from 18% to 57%.

In addition to math problems, chain of thought prompting also lifted performance on questions related to sports understanding, coin flip tracking, and last letter concatenation. In most cases, not many examples were need to saturate the performance gains (less than 8 or so).

To learn more, read the full paper.


Implications
One advantage of the few-shot example-based approach relative to the Let's think step by step technique is that you can more easily specify the format, length, and style of reasoning that you want the model to perform before landing on its final answer. This can be particularly helpful in cases where the model isn't initially reasoning in the right way or depth.


Method
In general, to eke out maximum performance on a task, you'll need to fine-tune a custom model. However, fine-tuning a model using explanations may take thousands of example explanations, which are costly to write.

In 2022, Eric Zelikman and Yuhuai Wu et al. published a clever procedure for using a few-shot prompt to generate a dataset of explanations that could be used to fine-tune a model. The idea is to use a few-shot prompt to generate candidate explanations, and only keep the explanations that produce the correct answer. Then, to get additional explanations for some of the incorrect answers, retry the few-shot prompt but with correct answers given as part of the question. The authors called their procedure STaR (Self-taught Reasoner):


Results
When the authors applied this technique to a Common Sense Q&A dataset, they found that STaR outperformed both chain-of-thought prompting alone (73% > 37%) and fine-tuning alone (73% > 60%):

STaR results
Source: STaR: Bootstrapping Reasoning With Reasoning by Eric Zelikman and Yujuai Wu et al. (2022)

To learn more, read the full paper.


Using a few-shot prompt to extend or modify a fine-tuning dataset is an idea that can be generalized beyond explanation writing. For example, if you have large quantities of unstructured text that you want to train on, you may find opportunities to use a prompt to extract a structured dataset from your unstructured text, and then fine-tune a custom model on that structured dataset.

Extensions to chain-of-thought prompting
A number of extensions of chain-of-thought prompting have been published as well.

Selection-inference prompting
Method
Published by Antonia Creswell et al., one extension of the chain-of-thought technique is to split the single prompt for generating explanations and answers into smaller parts. First, a prompt selects a relevant subset of facts from the text ('selection prompt'). Then, a second prompt infers a conclusion from the selected facts ('inference prompt'). These prompts are then alternated in a loop to generate multiple steps of reasoning and eventually land on a final answer. The authors illustrate the idea in the following figure:


Selection-inference prompting
Source: Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning by Antonia Creswell et al. (2022)

When applied to a 7B-parameter model, the authors found that selection-inference prompting substantially improved performance relative to chain-of-thought prompting on the bAbi and Proof Writer benchmark tasks (both of which require longer sequences of reasoning steps). The best performance they achieved combined both selection-inference prompting with fine-tuning.

Selection-inference prompting
Source: Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning by Antonia Creswell et al. (2022)


Although the gains on these benchmarks were large, these benchmarks were specifically chosen because they required longer sequences of reasoning. On problems that don't require reasoning with many steps, the gains are likely smaller.

The results highlight a couple of general lessons for working with large language models. One, splitting up complex tasks into smaller tasks is a great way to improve reliability and performance; the more atomic the task, the less room there is for the model to err. Two, getting maximum performance often means combining fine-tuning with whatever approach you've chosen.

To learn more, read the full paper.


A few months after publishing the selection-inference prompting technique, the authors extended the technique in a follow-up paper, with ideas for:

figuring out when the selection-inference cycle should stop or continue
adding a value function to help search over multiple reasoning paths
reducing hallucination of fake facts by fine-tuning a model to reason about sentence labels (e.g., sen1) rather than writing out the sentences themselves
Method
In the original selection-inference technique, specialized 'selection' and 'inference' prompts are alternated to select facts and make inferences from those facts, combining to generate a sequence of reasoning steps.


The authors evaluated their technique on two benchmarks: the ProofWriter task (not shown) and EntailmentBankQA (shown). The technique increased accuracy substantially, especially on harder reasoning problems.

Implications
This paper illustrates a number of helpful lessons for improving the reliability of large language models:

Split complex tasks into smaller, more reliable subtasks
Generate your answer in a step-by-step fashion, evaluating it along the way
Generate many possible answers and use another model or function to pick the ones that look best
Reduce hallucination by constraining what the model can say (e.g., by using sentence labels instead of sentences)
Maximize performance of models by fine-tuning them on specialized tasks
To learn more, read the full paper.


Least-to-most prompting
Source: Least-to-most Prompting Enables Complex Reasoning in Large Language Models by Denny Zhou et al. (2022)

Results
When applied to benchmarks involving long reasoning chains using code-davinci-002 (which is optimized for code but can still understand text), the authors measured gains as large as 16% -> 99.7%!

Implications
Although the above gains from least-to-most prompting are impressive, they are measured on a very narrow set of tasks that require long reasoning chains.

Still, they illustrate a common theme: increase reliability by (a) breaking complex tasks into smaller subtasks and (b) giving the model more time and space to work out the answer.

To learn more, read the full paper.


Maieutic prompting
Method
In contrast to the previous techniques, which try to maximize the likelihood of correct answers, another approach is to use GPT-3 to generate a tree of possible explanations (both correct and incorrect), and then analyze their relationships to guess at which set is correct. This technique was coined maieutic prompting by Jaehun Jung et al. in May 2022 (maieutic means relating to the Socratic method of asking questions to elicit ideas).

The method is complicated, and works as follows:

First, build a maieutic tree, where each node is a statement that could be true or false:
Start with a multiple-choice question or true/false statement (e.g. War cannot have a tie)
For each possible answer to the question, use the model to generate a corresponding explanation (with a prompt like War cannot have a tie? True, because)
Then, prompt the model with the question and the generated explanation, and ask it to produce the answer. If reversing the explanation (with a prefix like It is wrong to say that {explanation}) reverses the answer, then the explanation is considered 'logically integral.'


Maieutic prompting results
Source: Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations by Jaehun Jung et al. (2022)

Implications
Beyond the complexity, one limitation of this method is that it appears to only apply to questions that can be posed as multiple-choice.

To learn more, read the full paper.


Self-consistency
Method
For tasks with a discrete set of answers, one simple way to improve reliability is to sample multiple explanations & answers from the model (using a positive temperature) and then pick the final answer that appears most often.


This technique lifted accuracies by anywhere from 1 to 24 percentage points on a suite of math and reasoning benchmarks. (Plotted below are results from Google's LaMDA model; using Google's larger PaLM model, the baselines were higher but the gains were a bit smaller.)

Implications
Although this technique is simple to implement, it can be costly. Generating a set of 10 answers will increase your costs by 10x.

Also, as with many of these techniques, it applies only to tasks with a limited set of answers. For open-ended tasks where each answer is unique (such as writing a poem), it's not obvious what it would mean to pick the most common answer.

Lastly, this technique ought to be most beneficial when there are multiple paths or phrasings to reach an answer; if there's only one path, then the technique may not help at all. An extreme example: If the task was to generate a single token answer, then taking the most common token from 100 generations would be no different than taking the token with the highest logprobs (which you can get with a single generation at temperature=0).


Another key technique for improving task performance is to train a verifier or discriminator model to evaluate the outputs of the main generative model. If the discriminator rejects the output, then you can resample the generative model until you get an acceptable output. In many cases, it's easier to judge an answer than it is to create an answer, which helps explain the power of this method.

Method
In 2021, OpenAI researchers applied this technique to grade school math problems, using the following procedure:


With a 175B GPT-3 model and 8,000 training examples, this technique substantially lifted grade school math accuracy from ~33% to ~55%.

Implications
Similar to the self-consistency technique, this method can get expensive, as generating, say, 100 solutions per task will increase your costs by roughly ~100x.


Although the techniques above vary in their approach, they all share the goal of improving reliability on complex tasks. Mainly they do this by:

decomposing unreliable operations into smaller, more reliable operations (e.g., selection-inference prompting)
using multiple steps or multiple relationships to make the system's reliability greater than any individual component (e.g., maieutic prompting)
Probabilistic graphical models
This paradigm of trying to build a reliable system out of less reliable components is reminiscent of probabilistic programming, and many of the analysis techniques of that field can be applied to this one.

In the paper Language Model Cascades, David Dohan et al. interpret the above techniques in the paradigm of probabilistic graphical models:


Although formulating these techniques as probabilistic graphical models may not be immediately useful for solving any particular problem, the framework may be helpful in selecting, combining, and discovering new techniques.

Closing thoughts
Research into large language models is very active and evolving rapidly. Not only do researchers continue to improve the models, they also continue to improve our understanding of how to best employ the models. To underscore the pace of these developments, note that all of the papers shared above were published within the past 12 months (as I write in Sep 2022).

In the future, expect better models and better techniques to be published. Even if the specific techniques here are eclipsed by future best practices, the general principles behind them will likely remain a key part of any expert user's toolkit.


Large language models are functions that map text to text. Given an input string of text, a large language model predicts the text that should come next.

The magic of large language models is that by being trained to minimize this prediction error over vast quantities of text, the models end up learning concepts useful for these predictions. For example, they learn:

how to spell
how grammar works
how to paraphrase
how to answer questions
how to hold a conversation
how to write in many languages
how to code
etc.
None of these capabilities are explicitly programmed in—they all emerge as a result of training.


Of all the inputs to a large language model, by far the most influential is the text prompt.

Large language models can be prompted to produce output in a few ways:

Instruction: Tell the model what you want
Completion: Induce the model to complete the beginning of what you want
Demonstration: Show the model what you want, with either:
A few examples in the prompt
Many hundreds or thousands of examples in a fine-tuning training dataset
An example of each is shown below.


Instruction-following models (e.g., text-davinci-003 or any model beginning with text-) are specially designed to follow instructions. Write your instruction at the top of the prompt (or at the bottom, or both), and the model will do its best to follow the instruction and then stop. Instructions can be detailed, so don't be afraid to write a paragraph explicitly detailing the output you want.

Example instruction prompt:

Extract the name of the author from the quotation below.

“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
― Ted Chiang, Exhalation
Output:

Ted Chiang


Completion-style prompts take advantage of how large language models try to write text they think is mostly likely to come next. To steer the model, try beginning a pattern or sentence that will be completed by the output you want to see. Relative to direct instructions, this mode of steering large language models can take more care and experimentation. In addition, the models won't necessarily know where to stop, so you will often need stop sequences or post-processing to cut off text generated beyond the desired output.

Example completion prompt:

“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
― Ted Chiang, Exhalation

The author of this quote is
Output:

Ted Chiang


Demonstration prompt example (few-shot learning)

Similar to completion-style prompts, demonstrations can show the model what you want it to do. This approach is sometimes called few-shot learning, as the model learns from a few examples provided in the prompt.

Example demonstration prompt:

Quote:
“When the reasoning mind is forced to confront the impossible again and again, it has no choice but to adapt.”
― N.K. Jemisin, The Fifth Season
Author: N.K. Jemisin

Quote:
“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
― Ted Chiang, Exhalation
Author:
Output:

Ted Chiang


With enough training examples, you can fine-tune a custom model. In this case, instructions become unnecessary, as the model can learn the task from the training data provided. However, it can be helpful to include separator sequences (e.g., -> or ### or any string that doesn't commonly appear in your inputs) to tell the model when the prompt has ended and the output should begin. Without separator sequences, there is a risk that the model continues elaborating on the input text rather than starting on the answer you want to see.

Example fine-tuned prompt (for a model that has been custom trained on similar prompt-completion pairs):

“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
― Ted Chiang, Exhalation
###
Output:

Ted Chiang


Large language models aren't only great at text - they can be great at code too. OpenAI's specialized code model is called Codex.

Codex powers more than 70 products, including:

GitHub Copilot (autocompletes code in VS Code and other IDEs)
Pygma (turns Figma designs into code)
Replit (has an 'Explain code' button and other features)
Warp (a smart terminal with AI command search)
Machinet (writes Java unit test templates)
Note that unlike instruction-following text models (e.g., text-davinci-002), Codex is not trained to follow instructions. As a result, designing good prompts can take more care.


For more prompt examples, visit OpenAI Examples.

In general, the input prompt is the best lever for improving model outputs. You can try tricks like:

Give more explicit instructions. E.g., if you want the output to be a comma separated list, ask it to return a comma separated list. If you want it to say "I don't know" when it doesn't know the answer, tell it 'Say "I don't know" if you do not know the answer.'

Supply better examples. If you're demonstrating examples in your prompt, make sure that your examples are diverse and high quality.

Ask the model to answer as if it was an expert. Explicitly asking the model to produce high quality output or output as if it was written by an expert can induce the model to give higher quality answers that it thinks an expert would write. E.g., "The following answer is correct, high-quality, and written by an expert."

Prompt the model to write down the series of steps explaining its reasoning. E.g., prepend your answer with something like "Let's think step by step." Prompting the model to give an explanation of its reasoning before its final answer can increase the likelihood that its final answer is consistent and correct.


This notebook provides a step-by-step guide for our new gpt-3.5-turbo fine-tuning. We'll perform entity extraction using the RecipeNLG dataset, which provides various recipes and a list of extracted generic ingredients for each. This is a common dataset for named entity recognition (NER) tasks.


We will go through the following steps:

Setup: Loading our dataset and filtering down to one domain to fine-tune on.
Data preparation: Preparing your data for fine-tuning by creating training and validation examples, and uploading them to the Files endpoint.


Fine-tuning: Creating your fine-tuned model.
Inference: Using your fine-tuned model for inference on new inputs.
By the end of this you should be able to train, evaluate and deploy a fine-tuned gpt-3.5-turbo model.


For more information on fine-tuning, you can refer to our documentation guide, API reference or blog post

Setup
# make sure to use the latest version of the openai python package
!pip install --upgrade openai

import json
import openai
import os
import pandas as pd
from pprint


OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "")

Fine-tuning works best when focused on a particular domain. It's important to make sure your dataset is both focused enough for the model to learn, but general enough that unseen examples won't be missed. Having this in mind, we have extracted a subset from the RecipesNLG dataset to only contain documents from www.cookbooks.com.


# Read in the dataset we'll use for this task.
# This will be the RecipesNLG dataset, which we've cleaned to only contain documents from www.cookbooks.com
recipe_df = pd.read_csv("data/cookbook_recipes_nlg_10k.csv")

recipe_df.head()


Data preparation
We'll begin by preparing our data. When fine-tuning with the ChatCompletion format, each training example is a simple list of messages. For example, an entry could look like:


[{'role': 'system',
  'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'},


{'role': 'user',
  'content': 'Title: No-Bake Nut Cookies\n\nIngredients: ["1 c. firmly packed brown sugar", "1/2 c. evaporated milk", "1/2 tsp. vanilla", "1/2 c. broken nuts (pecans)", "2 Tbsp. butter or margarine", "3 1/2 c. bite size shredded rice biscuits"]\n\nGeneric ingredients: '},


{'role': 'assistant',
  'content': '["brown sugar", "milk", "vanilla", "nuts", "butter", "bite size shredded rice biscuits"]'}]


During the training process this conversation will be split, with the final entry being the completion that the model will produce, and the remainder of the messages acting as the prompt. Consider this when building your training examples - if your model will act on multi-turn conversations, then please provide representative examples so it doesn't perform poorly when the conversation starts to expand.


Please note that currently there is a 4096 token limit for each training example. Anything longer than this will be truncated at 4096 tokens.


training_data = []

system_message = "You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided."


def create_user_message(row):
    return f"Title: {row['title']}\n\nIngredients: {row['ingredients']}\n\nGeneric ingredients: "


def prepare_example_conversation(row):
    messages = []
    messages.append({"role": "system", "content": system_message})


user_message = create_user_message(row)
    messages.append({"role": "user", "content": user_message})

    messages.append({"role": "assistant", "content": row["NER"]})


return {"messages": messages}

pprint(prepare_example_conversation(recipe_df.iloc[0]))


Let's now do this for a subset of the dataset to use as our training data. You can begin with even 30-50 well-pruned examples. You should see performance continue to scale linearly as you increase the size of the training set, but your jobs will also take longer.


# use the first 100 rows of the dataset for training
training_df = recipe_df.loc[0:100]


# apply the prepare_example_conversation function to each row of the training_df
training_data = training_df.apply(prepare_example_conversation, axis=1).tolist()


for example in training_data[:5]:
    print(example)


In addition to training data, we can also optionally provide validation data, which will be used to make sure that the model does not overfit your training set.


validation_df = recipe_df.loc[101:200]
validation_data = validation_df.apply(prepare_example_conversation, axis=1).tolist()


We then need to save our data as .jsonl files, with each line being one training example conversation.


def write_jsonl(data_list: list, filename: str) -> None:
    with open(filename, "w") as out:
        for ddict in data_list:
            jout = json.dumps(ddict) + "\n"
            out.write(jout)


training_file_name = "tmp_recipe_finetune_training.jsonl"
write_jsonl(training_data, training_file_name)


validation_file_name = "tmp_recipe_finetune_validation.jsonl"
write_jsonl(validation_data, validation_file_name)


This is what the first 5 lines of our training .jsonl file look like:

# print the first 5 lines of the training file
!head -n 5 tmp_recipe_finetune_training.jsonl


Upload files
You can now upload the files to our Files endpoint to be used by the fine-tuned model.


training_response = openai.File.create(
    file=open(training_file_name, "rb"), purpose="fine-tune"
)
training_file_id = training_response["id"]

validation_response = openai.File.create(
    file=open(validation_file_name, "rb"), purpose="fine-tune"
)
validation_file_id = validation_response["id"]


print("Training file ID:", training_file_id)
print("Validation file ID:", validation_file_id)


Fine-tuning
Now we can create our fine-tuning job with the generated files and an optional suffix to identify the model. The response will contain an id which you can use to retrieve updates on the job.


Note: The files have to first be processed by our system, so you might get a File not ready error. In that case, simply retry a few minutes later.


response = openai.FineTuningJob.create(
    training_file=training_file_id,
    validation_file=validation_file_id,
    model="gpt-3.5-turbo",
    suffix="recipe-ner",
)


job_id = response["id"]

print("Job ID:", response["id"])
print("Status:", response["status"])


Check job status
You can make a GET request to the https://api.openai.com/v1/alpha/fine-tunes endpoint to list your alpha fine-tune jobs. In this instance you'll want to check that the ID you got from the previous step ends up as status: succeeded.


Once it is completed, you can use the result_files to sample the results from the validation set (if you uploaded one), and use the ID from the fine_tuned_model parameter to invoke your trained model.


response = openai.FineTuningJob.retrieve(job_id)

print("Job ID:", response["id"])


print("Status:", response["status"])
print("Trained Tokens:", response["trained_tokens"])


We can track the progress of the fine-tune with the events endpoint. You can rerun the cell below a few times until the fine-tune is ready.


response = openai.FineTuningJob.list_events(id=job_id, limit=50)

events = response["data"]
events.reverse()


for event in events:
    print(event["message"])


Now that it's done, we can get a fine-tuned model ID from the job:

response = openai.FineTuningJob.retrieve(job_id)
fine_tuned_model_id = response["fine_tuned_model"]

if fine_tuned_model_id is None:
    raise RuntimeError("Fine-tuned model ID not found. Your job has likely not been completed yet.")


print("Fine-tuned model ID:", fine_tuned_model_id)


Inference
The last step is to use your fine-tuned model for inference. Similar to the classic FineTuning, you simply call ChatCompletions with your new fine-tuned model name filling the model parameter.


test_df = recipe_df.loc[201:300]
test_row = test_df.iloc[0]
test_messages = []
test_messages.append({"role": "system", "content": system_message})


user_message = create_user_message(test_row)
test_messages.append({"role": "user", "content": create_user_message(test_row)})


pprint(test_messages)
response = openai.ChatCompletion.create(
    model=fine_tuned_model_id, messages=test_messages, temperature=0, max_tokens=500
)


Conclusion
Congratulations, you are now ready to fine-tune your own models using the ChatCompletion format! We look forward to seeing what you build


Evaluating Abstractive Summarization - Part 1

In this notebook we delve into the evaluation techniques for abstractive summarization tasks using a simple example. We explore traditional evaluation methods like ROUGE and BERTScore, in addition to showcasing a more novel approach using LLMs as evaluators.


Evaluating Abstractive Summarization - Part 2

Evaluating the quality of summaries is a time-consuming process, as it involves different quality metrics such as coherence, conciseness, readability and content. Traditional automatic evaluation metrics such as ROUGE and BERTScore and others are concrete and reliable, but they may not correlate well with the actual quality of summaries. They show relatively low correlation with human judgments, especially for open-ended generation tasks (Liu et al., 2023). There's a growing need to lean on human evaluations, user feedback, or model-based metrics while being vigilant about potential biases. While human judgment provides invaluable insights, it is often not scalable and can be cost-prohibitive.


Evaluating Abstractive Summarization - Part 3

In addition to these traditional metrics, we showcase a method (G-Eval) that leverages Large Language Models (LLMs) as a novel, reference-free metric for assessing abstractive summaries. In this case, we use gpt-4 to score candidate outputs. gpt-4 has effectively learned an internal model of language quality that allows it to differentiate between fluent, coherent text and low-quality text. Harnessing this internal scoring mechanism allows auto-evaluation of new candidate outputs generated by an LLM.


Evaluating Abstractive Summarization - Part 4

# Installing necessary packages for the evaluation
# rouge: For evaluating with ROUGE metric
# bert_score: For evaluating with BERTScore
# openai: To interact with OpenAI's API
!pip install rouge --quiet
!pip install bert_score --quiet
!pip install openai --quiet


Evaluating Abstractive Summarization - Part 5

import openai
import os
import re
import pandas as pd

# Python Implementation of the ROUGE Metric
from rouge import Rouge

# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.
from bert_score import BERTScorer

openai.api_key = os.environ.get("OPENAI_API_KEY")


Evaluating Abstractive Summarization - Part 6

For the purposes of this notebook we'll use the example summarization below. Notice that we provide two generated summaries to compare, and a reference human-written summary, which evaluation metrics like ROUGE and BERTScore require.


Evaluating Abstractive Summarization - Part 7

Excerpt (excerpt):

OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.


Evaluating Abstractive Summarization - Part 8

Summaries:

Reference Summary /ref_summary (human generated)
Eval Summary 1 / eval_summary_1 (system generated)
Eval Summary 2 / eval_summary_2 (system generated)


Evaluating Abstractive Summarization - Part 9

Take a moment to figure out which summary you'd personally prefer and the one that captures OpenAI's mission really well.


Evaluating Abstractive Summarization (Chunk 1)

OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.


Evaluating Abstractive Summarization (Chunk 2)

ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, ROUGE-L offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.


Evaluating Abstractive Summarization (Chunk 3)

The table shows the ROUGE scores for evaluating two different summaries against a reference text. In the case of rouge-1, Summary 2 outperforms Summary 1, indicating a better overlap of individual words and for rouge-l, Summary 2 has a higher score, implying a closer match in the longest common subsequences, and thus a potentially better overall summarization in capturing the main content and order of the original text. Since Summary 2 has many words and short phrases directly lifted from the excerpt, its overlap with the reference summary would likely be higher, leading to higher ROUGE scores.


Evaluating Abstractive Summarization (Chunk 4)

While ROUGE and similar metrics, such as BLEU and METEOR, offer quantitative measures, they often fail to capture the true essence of a well-generated summary. They also correlate worse with human scores. Given the advancements in LLMs, which are adept at producing fluent and coherent summaries, traditional metrics like ROUGE may inadvertently penalize these models. This is especially true if the summaries are articulated differently but still encapsulate the core information accurately.


Evaluating Abstractive Summarization (Chunk 5)

Evaluating using BERTScore
ROUGE relies on the exact presence of words in both the predicted and reference texts, failing to interpret the underlying semantics. This is where BERTScore comes in and leverages the contextual embeddings from the BERT model, aiming to evaluate the similarity between a predicted and a reference sentence in the context of machine-generated text. By comparing embeddings from both sentences, BERTScore captures semantic similarities that might be missed by traditional n-gram based metrics.


Evaluating Abstractive Summarization (Chunk 6)

The close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since BERTScore may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining BERTScore with human judgment and other metrics could offer a more reliable evaluation.


Evaluating Abstractive Summarization (Chunk 7)

Evaluating using GPT-4
Here we implement an example reference-free text evaluator using gpt-4, inspired by the G-Eval framework which evaluates the quality of generated text using large language models. Unlike metrics like ROUGE or BERTScore that rely on comparison to reference summaries, the gpt-4 based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable.


Evaluating Abstractive Summarization (Chunk 8)

In this demonstration, we're using a direct scoring function where gpt-4 generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.


Evaluating Abstractive Summarization - Part 1

Overall, the Summary 1 appears to outperform Summary 2 in three of the four categories (Coherence, Relevance and Fluency). Both summaries are found to be consistent with each other. The result might suggest that Summary 1 is generally preferable based on the given evaluation criteria.


Evaluating Abstractive Summarization - Part 2

Limitations
Note that LLM-based metrics could have a bias towards preferring LLM-generated texts over human-written texts. Additionally LLM based metrics are sensitive to system messages/prompts. We recommend experimenting with other techniques that can help improve performance and/or get consistent scores, striking the right balance between high-quality expensive evaluation and automated evaluations. It is also worth noting that this scoring methodology is currently limited by gpt-4's context window.


Evaluating Abstractive Summarization - Part 3

Conclusion
Evaluating abstractive summarization remains an open area for further improvement. Traditional metrics like ROUGE, BLEU, and BERTScore provide useful automatic evaluation but have limitations in capturing semantic similarity and nuanced aspects of summarization quality. Moreover, they require reference outputs which can be expensive to collect/label. LLM-based metrics offer promise as a reference-free method of evaluating coherence, fluency, and relevance. However, they too have potential biases favoring text generated by LLMs. Ultimately, a combination of automatic metrics and human evaluation is ideal for reliably assessing abstractive summarization systems. While human evaluation is indispensable for gaining a comprehensive understanding of summary quality, it should be complemented with automated evaluation to enable efficient, large-scale testing. The field will continue to evolve more robust evaluation techniques, balancing quality, scalability, and fairness. Advancing evaluation methods is crucial for driving progress in production applications.


Question answering using a search API and re-ranking

Searching for relevant information can sometimes feel like looking for a needle in a haystack, but don’t despair, GPTs can actually do a lot of this work for us. In this guide we explore a way to augment existing search systems with various AI techniques, helping us sift through the noise.


Two ways of retrieving information for GPT are:

Mimicking Human Browsing: GPT triggers a search, evaluates the results, and modifies the search query if necessary. It can also follow up on specific search results to form a chain of thought, much like a human user would do.


Retrieval with Embeddings: Calculate embeddings for your content and a user query, and then retrieve the content most related as measured by cosine similarity. This technique is used heavily by search engines like Google.


By combining these approaches, and drawing inspiration from re-ranking methods, we identify an approach that sits in the middle. This approach can be implemented on top of any existing search system, like the Slack search API, or an internal ElasticSearch instance with private data. Here’s how it works:


Step 1: Search
User asks a question.
GPT generates a list of potential queries.
Search queries are executed in parallel.


Step 2: Re-rank
Embeddings for each result are used to calculate semantic similarity to a generated hypothetical ideal answer to the user question.
Results are ranked and filtered based on this similarity metric.


Step 3: Answer
Given the top search results, the model generates an answer to the user’s question, including references and links.
This hybrid approach offers relatively low latency and can be integrated into any existing search endpoint, without requiring the upkeep of a vector database. Let's dive into it! We will use the News API as an example domain to search over.


Setup
In addition to your OPENAI_API_KEY, you'll have to include a NEWS_API_KEY in your environment. You can get an API key here.


User asks a question. GPT generates a list of potential queries. Search queries are executed in parallel.


Re-rank
Drawing inspiration from HyDE (Gao et al.), we first generate a hypothetical ideal answer to rerank our compare our results against. This helps prioritize results that look like good answers, rather than those similar to our question. Here’s the prompt we use to generate our hypothetical answer.


Generate a hypothetical answer to the user's question. This answer will be used to rank search results. Pretend you have all the information you need to answer, but don't use any actual facts. Instead, use placeholders like NAME did something, or NAME said something at PLACE.


Now, let's generate embeddings for the search results and the hypothetical answer. We then calculate the cosine distance between these embeddings, giving us a semantic similarity metric. Note that we can simply calculate the dot product in lieu of doing a full cosine similarity calculation since the OpenAI embeddings are returned normalized in our API.


Finally, we use these similarity scores to sort and filter the results.


These results look a lot more relevant to our original query. Now, let's use the top 5 results to generate a final answer.


Generate an answer to the user's question based on the given search results. TOP_RESULTS: [{'title': 'Article Title 1', 'description': 'Article Description 1', 'url': 'https://example.com/article1'}, ...] USER_QUESTION: Who won the NBA championship? And who was the MVP? Tell me a bit about the last game.


Now, in order to be as exhaustive as possible, we use the model to generate a list of diverse queries based on this question. QUERIES_INPUT = f"""
You have access to a search API that returns recent news articles. Generate an array of search queries that are relevant to this question. Use a variation of related keywords for the queries, trying to be as general as possible. Include as many queries as you can think of, including and excluding terms. For example, include queries like ['keyword_1 keyword_2', 'keyword_1', 'keyword_2']. Be creative. The more queries you include, the more likely you are to find relevant results. User question: {USER_QUESTION} Format: {"queries": ["query_1", "query_2", "query_3"]} queries = json_gpt(QUERIES_INPUT)["queries"] # Let's include the original question as well for good measure queries.append(USER_QUESTION) queries The queries look good, so let's run the searches.


As we can see, oftentimes, the search queries will return a large number of results, many of which are not relevant to the original question asked by the user. In order to improve the quality of the final answer, we use embeddings to re-rank and filter the results. 2. Re-rank Drawing inspiration from HyDE (Gao et al.), we first generate a hypothetical ideal answer to rerank our compare our results against. This helps prioritize results that look like good answers, rather than those similar to our question. Here’s the prompt we use to generate our hypothetical answer. HA_INPUT = f"""
Generate a hypothetical answer to the user's question. This answer will be used to rank search results. Pretend you have all the information you need to answer, but don't use any actual facts. Instead, use placeholders like NAME did something, or NAME said something at PLACE. User question: {USER_QUESTION} Format: {"hypotheticalAnswer": "hypothetical answer text"} hypothetical_answer = json_gpt(HA_INPUT)["hypotheticalAnswer"] hypothetical_answer Now, let's generate embeddings for the search results and the hypothetical answer. We then calculate the cosine distance between these embeddings, giving us a semantic similarity metric. Note that we can simply calculate the dot product in lieu of doing a full cosine similarity calculation since the OpenAI embeddings are returned normalized in our API.