How do i get the confidence score of an answer? I want to use this model for question answering task.
How do i get the confidence score of an answer? I want to use this model for question answering task.
If the question is such that the answers are a kind of mutually exclusive multiple choice answer, with a single token answer, then you can get the token probability of the answer. This is available in the API, but not the python API. You can request the top K highest probability tokens and look at the probability of the generated answer token.
The python API has this in the old openai.Completion interface, which works with some older models only.
response = openai.Completion.create(
model="text-davinci-002",
prompt=prompt,
max_tokens=7,
logprobs=5,
temperature=0
)
Oops you can ignore my comments about the openai API. You can get the token probabilities for the mistral huggingface output.