Inference
Inference is the process of using a trained model to make predictions on new data. As this process can be compute-intensive,
running on a dedicated server can be an interesting option. The huggingface_hub library provides an easy way to call a
service that runs inference for hosted models. There are several services you can connect to:
- Inference API: a service that allows you to run accelerated inference on Hugging Face’s infrastructure for free. This service is a fast way to get started, test different models, and prototype AI products.
- Inference Endpoints: a product to easily deploy models to production. Inference is run by Hugging Face in a dedicated, fully managed infrastructure on a cloud provider of your choice.
These services can be called with the InferenceClient object. Please refer to this guide for more information on how to use it.
Inference Client
class huggingface_hub.InferenceClient
< source >( model: typing.Optional[str] = None token: typing.Union[str, bool, NoneType] = None timeout: typing.Optional[float] = None headers: typing.Optional[typing.Dict[str, str]] = None cookies: typing.Optional[typing.Dict[str, str]] = None proxies: typing.Optional[typing.Any] = None base_url: typing.Optional[str] = None api_key: typing.Optional[str] = None )
Parameters
- model (
str,optional) — The model to run inference with. Can be a model id hosted on the Hugging Face Hub, e.g.meta-llama/Meta-Llama-3-8B-Instructor a URL to a deployed Inference Endpoint. Defaults to None, in which case a recommended model is automatically selected for the task. Note: for better compatibility with OpenAI’s client,modelhas been aliased asbase_url. Those 2 arguments are mutually exclusive. If usingbase_urlfor chat completion, the/chat/completionssuffix path will be appended to the base URL (see the TGI Messages API documentation for details). When passing a URL asmodel, the client will not append any suffix path to it. - token (
strorbool, optional) — Hugging Face token. Will default to the locally saved token if not provided. Passtoken=Falseif you don’t want to send your token to the server. Note: for better compatibility with OpenAI’s client,tokenhas been aliased asapi_key. Those 2 arguments are mutually exclusive and have the exact same behavior. - timeout (
float,optional) — The maximum number of seconds to wait for a response from the server. Loading a new model in Inference API can take up to several minutes. Defaults to None, meaning it will loop until the server is available. - headers (
Dict[str, str],optional) — Additional headers to send to the server. By default only the authorization and user-agent headers are sent. Values in this dictionary will override the default values. - cookies (
Dict[str, str],optional) — Additional cookies to send to the server. - proxies (
Any,optional) — Proxies to use for the request. - base_url (
str,optional) — Base URL to run inference. This is a duplicated argument frommodelto make InferenceClient follow the same pattern asopenai.OpenAIclient. Cannot be used ifmodelis set. Defaults to None. - api_key (
str,optional) — Token to use for authentication. This is a duplicated argument fromtokento make InferenceClient follow the same pattern asopenai.OpenAIclient. Cannot be used iftokenis set. Defaults to None.
Initialize a new Inference Client.
InferenceClient aims to provide a unified experience to perform inference. The client can be used seamlessly with either the (free) Inference API or self-hosted Inference Endpoints.
audio_classification
< source >( audio: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None top_k: typing.Optional[int] = None function_to_apply: typing.Optional[ForwardRef('AudioClassificationOutputTransform')] = None ) → List[AudioClassificationOutputElement]
Parameters
- audio (Union[str, Path, bytes, BinaryIO]) — The audio content to classify. It can be raw audio bytes, a local audio file, or a URL pointing to an audio file.
- model (
str, optional) — The model to use for audio classification. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for audio classification will be used. - top_k (
int, optional) — When specified, limits the output to the top K most probable classes. - function_to_apply (
"AudioClassificationOutputTransform", optional) — The function to apply to the output.
Returns
List[AudioClassificationOutputElement]
List of AudioClassificationOutputElement items containing the predicted labels and their confidence.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Perform audio classification on the provided audio content.
audio_to_audio
< source >( audio: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None ) → List[AudioToAudioOutputElement]
Parameters
- audio (Union[str, Path, bytes, BinaryIO]) — The audio content for the model. It can be raw audio bytes, a local audio file, or a URL pointing to an audio file.
- model (
str, optional) — The model can be any model which takes an audio file and returns another audio file. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for audio_to_audio will be used.
Returns
List[AudioToAudioOutputElement]
A list of AudioToAudioOutputElement items containing audios label, content-type, and audio content in blob.
Raises
InferenceTimeoutError or HTTPError
InferenceTimeoutError— If the model is unavailable or the request times out.HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Performs multiple tasks related to audio-to-audio depending on the model (eg: speech enhancement, source separation).
automatic_speech_recognition
< source >( audio: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None ) → AutomaticSpeechRecognitionOutput
Parameters
- audio (Union[str, Path, bytes, BinaryIO]) — The content to transcribe. It can be raw audio bytes, local audio file, or a URL to an audio file.
- model (
str, optional) — The model to use for ASR. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for ASR will be used.
Returns
An item containing the transcribed text and optionally the timestamp chunks.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Perform automatic speech recognition (ASR or audio-to-text) on the given audio content.
chat_completion
< source >( messages: typing.List[typing.Dict] model: typing.Optional[str] = None stream: bool = False frequency_penalty: typing.Optional[float] = None logit_bias: typing.Optional[typing.List[float]] = None logprobs: typing.Optional[bool] = None max_tokens: typing.Optional[int] = None n: typing.Optional[int] = None presence_penalty: typing.Optional[float] = None response_format: typing.Optional[huggingface_hub.inference._generated.types.chat_completion.ChatCompletionInputGrammarType] = None seed: typing.Optional[int] = None stop: typing.Optional[typing.List[str]] = None stream_options: typing.Optional[huggingface_hub.inference._generated.types.chat_completion.ChatCompletionInputStreamOptions] = None temperature: typing.Optional[float] = None tool_choice: typing.Union[huggingface_hub.inference._generated.types.chat_completion.ChatCompletionInputToolType, str, NoneType] = None tool_prompt: typing.Optional[str] = None tools: typing.Optional[typing.List[huggingface_hub.inference._generated.types.chat_completion.ToolElement]] = None top_logprobs: typing.Optional[int] = None top_p: typing.Optional[float] = None ) → ChatCompletionOutput or Iterable of ChatCompletionStreamOutput
Parameters
- messages (List of ChatCompletionInputMessage) — Conversation history consisting of roles and content pairs.
- model (
str, optional) — The model to use for chat-completion. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for chat-based text-generation will be used. See https://huggingface.co/tasks/text-generation for more details.If
modelis a model ID, it is passed to the server as themodelparameter. If you want to define a custom URL while settingmodelin the request payload, you must setbase_urlwhen initializing InferenceClient. - frequency_penalty (
float, optional) — Penalizes new tokens based on their existing frequency in the text so far. Range: [-2.0, 2.0]. Defaults to 0.0. - logit_bias (
List[float], optional) — Modify the likelihood of specified tokens appearing in the completion. Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token. Defaults to None. - logprobs (
bool, optional) — Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. - max_tokens (
int, optional) — Maximum number of tokens allowed in the response. Defaults to 100. - n (
int, optional) — UNUSED. - presence_penalty (
float, optional) — Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics. - response_format (ChatCompletionInputGrammarType, optional) — Grammar constraints. Can be either a JSONSchema or a regex.
- seed (Optional
int, optional) — Seed for reproducible control flow. Defaults to None. - stop (Optional
str, optional) — Up to four strings which trigger the end of the response. Defaults to None. - stream (
bool, optional) — Enable realtime streaming of responses. Defaults to False. - stream_options (ChatCompletionInputStreamOptions, optional) — Options for streaming completions.
- temperature (
float, optional) — Controls randomness of the generations. Lower values ensure less random completions. Range: [0, 2]. Defaults to 1.0. - top_logprobs (
int, optional) — An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used. - top_p (
float, optional) — Fraction of the most likely next words to sample from. Must be between 0 and 1. Defaults to 1.0. - tool_choice (ChatCompletionInputToolType or
str, optional) — The tool to use for the completion. Defaults to “auto”. - tool_prompt (
str, optional) — A prompt to be appended before the tools. - tools (List of ToolElement, optional) — A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for.
Returns
ChatCompletionOutput or Iterable of ChatCompletionStreamOutput
Generated text returned from the server:
- if
stream=False, the generated text is returned as a ChatCompletionOutput (default). - if
stream=True, the generated text is returned token by token as a sequence of ChatCompletionStreamOutput.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
A method for completing conversations using a specified language model.
The client.chat_completion method is aliased as client.chat.completions.create for compatibility with OpenAI’s client.
Inputs and outputs are strictly the same and using either syntax will yield the same results.
Check out the Inference guide
for more details about OpenAI’s compatibility.
Example:
>>> from huggingface_hub import InferenceClient
>>> messages = [{"role": "user", "content": "What is the capital of France?"}]
>>> client = InferenceClient("meta-llama/Meta-Llama-3-8B-Instruct")
>>> client.chat_completion(messages, max_tokens=100)
ChatCompletionOutput(
choices=[
ChatCompletionOutputComplete(
finish_reason='eos_token',
index=0,
message=ChatCompletionOutputMessage(
role='assistant',
content='The capital of France is Paris.',
name=None,
tool_calls=None
),
logprobs=None
)
],
created=1719907176,
id='',
model='meta-llama/Meta-Llama-3-8B-Instruct',
object='text_completion',
system_fingerprint='2.0.4-sha-f426a33',
usage=ChatCompletionOutputUsage(
completion_tokens=8,
prompt_tokens=17,
total_tokens=25
)
)Example using streaming:
>>> from huggingface_hub import InferenceClient
>>> messages = [{"role": "user", "content": "What is the capital of France?"}]
>>> client = InferenceClient("meta-llama/Meta-Llama-3-8B-Instruct")
>>> for token in client.chat_completion(messages, max_tokens=10, stream=True):
... print(token)
ChatCompletionStreamOutput(choices=[ChatCompletionStreamOutputChoice(delta=ChatCompletionStreamOutputDelta(content='The', role='assistant'), index=0, finish_reason=None)], created=1710498504)
ChatCompletionStreamOutput(choices=[ChatCompletionStreamOutputChoice(delta=ChatCompletionStreamOutputDelta(content=' capital', role='assistant'), index=0, finish_reason=None)], created=1710498504)
(...)
ChatCompletionStreamOutput(choices=[ChatCompletionStreamOutputChoice(delta=ChatCompletionStreamOutputDelta(content=' may', role='assistant'), index=0, finish_reason=None)], created=1710498504)Example using OpenAI’s syntax:
# instead of `from openai import OpenAI`
from huggingface_hub import InferenceClient
# instead of `client = OpenAI(...)`
client = InferenceClient(
base_url=...,
api_key=...,
)
output = client.chat.completions.create(
model="meta-llama/Meta-Llama-3-8B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Count to 10"},
],
stream=True,
max_tokens=1024,
)
for chunk in output:
print(chunk.choices[0].delta.content)Example using Image + Text as input:
>>> from huggingface_hub import InferenceClient
# provide a remote URL
>>> image_url ="https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
# or a base64-encoded image
>>> image_path = "/path/to/image.jpeg"
>>> with open(image_path, "rb") as f:
... base64_image = base64.b64encode(f.read()).decode("utf-8")
>>> image_url = f"data:image/jpeg;base64,{base64_image}"
>>> client = InferenceClient("meta-llama/Llama-3.2-11B-Vision-Instruct")
>>> output = client.chat.completions.create(
... messages=[
... {
... "role": "user",
... "content": [
... {
... "type": "image_url",
... "image_url": {"url": image_url},
... },
... {
... "type": "text",
... "text": "Describe this image in one sentence.",
... },
... ],
... },
... ],
... )
>>> output
The image depicts the iconic Statue of Liberty situated in New York Harbor, New York, on a clear day.Example using tools:
>>> client = InferenceClient("meta-llama/Meta-Llama-3-70B-Instruct")
>>> messages = [
... {
... "role": "system",
... "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.",
... },
... {
... "role": "user",
... "content": "What's the weather like the next 3 days in San Francisco, CA?",
... },
... ]
>>> tools = [
... {
... "type": "function",
... "function": {
... "name": "get_current_weather",
... "description": "Get the current weather",
... "parameters": {
... "type": "object",
... "properties": {
... "location": {
... "type": "string",
... "description": "The city and state, e.g. San Francisco, CA",
... },
... "format": {
... "type": "string",
... "enum": ["celsius", "fahrenheit"],
... "description": "The temperature unit to use. Infer this from the users location.",
... },
... },
... "required": ["location", "format"],
... },
... },
... },
... {
... "type": "function",
... "function": {
... "name": "get_n_day_weather_forecast",
... "description": "Get an N-day weather forecast",
... "parameters": {
... "type": "object",
... "properties": {
... "location": {
... "type": "string",
... "description": "The city and state, e.g. San Francisco, CA",
... },
... "format": {
... "type": "string",
... "enum": ["celsius", "fahrenheit"],
... "description": "The temperature unit to use. Infer this from the users location.",
... },
... "num_days": {
... "type": "integer",
... "description": "The number of days to forecast",
... },
... },
... "required": ["location", "format", "num_days"],
... },
... },
... },
... ]
>>> response = client.chat_completion(
... model="meta-llama/Meta-Llama-3-70B-Instruct",
... messages=messages,
... tools=tools,
... tool_choice="auto",
... max_tokens=500,
... )
>>> response.choices[0].message.tool_calls[0].function
ChatCompletionOutputFunctionDefinition(
arguments={
'location': 'San Francisco, CA',
'format': 'fahrenheit',
'num_days': 3
},
name='get_n_day_weather_forecast',
description=None
)Example using response_format:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient("meta-llama/Meta-Llama-3-70B-Instruct")
>>> messages = [
... {
... "role": "user",
... "content": "I saw a puppy a cat and a raccoon during my bike ride in the park. What did I saw and when?",
... },
... ]
>>> response_format = {
... "type": "json",
... "value": {
... "properties": {
... "location": {"type": "string"},
... "activity": {"type": "string"},
... "animals_seen": {"type": "integer", "minimum": 1, "maximum": 5},
... "animals": {"type": "array", "items": {"type": "string"}},
... },
... "required": ["location", "activity", "animals_seen", "animals"],
... },
... }
>>> response = client.chat_completion(
... messages=messages,
... response_format=response_format,
... max_tokens=500,
)
>>> response.choices[0].message.content
'{
y": "bike ride",
": ["puppy", "cat", "raccoon"],
_seen": 3,
n": "park"}'document_question_answering
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] question: str model: typing.Optional[str] = None doc_stride: typing.Optional[int] = None handle_impossible_answer: typing.Optional[bool] = None lang: typing.Optional[str] = None max_answer_len: typing.Optional[int] = None max_question_len: typing.Optional[int] = None max_seq_len: typing.Optional[int] = None top_k: typing.Optional[int] = None word_boxes: typing.Optional[typing.List[typing.Union[typing.List[float], str]]] = None ) → List[DocumentQuestionAnsweringOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image for the context. It can be raw bytes, an image file, or a URL to an online image. - question (
str) — Question to be answered. - model (
str, optional) — The model to use for the document question answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended document question answering model will be used. Defaults to None. - doc_stride (
int, optional) — If the words in the document are too long to fit with the question for the model, it will be split in several chunks with some overlap. This argument controls the size of that overlap. - handle_impossible_answer (
bool, optional) — Whether to accept impossible as an answer - lang (
str, optional) — Language to use while running OCR. Defaults to english. - max_answer_len (
int, optional) — The maximum length of predicted answers (e.g., only answers with a shorter length are considered). - max_question_len (
int, optional) — The maximum length of the question after tokenization. It will be truncated if needed. - max_seq_len (
int, optional) — The maximum length of the total sentence (context + question) in tokens of each chunk passed to the model. The context will be split in several chunks (using doc_stride as overlap) if needed. - top_k (
int, optional) — The number of answers to return (will be chosen by order of likelihood). Can return less than top_k answers if there are not enough options available within the context. - word_boxes (
List[Union[List[float], str, optional) — A list of words and bounding boxes (normalized 0->1000). If provided, the inference will skip the OCR step and use the provided bounding boxes instead.
Returns
List[DocumentQuestionAnsweringOutputElement]
a list of DocumentQuestionAnsweringOutputElement items containing the predicted label, associated probability, word ids, and page number.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Answer questions on document images.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.document_question_answering(image="https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png", question="What is the invoice number?")
[DocumentQuestionAnsweringOutputElement(answer='us-001', end=16, score=0.9999666213989258, start=16, words=None)]feature_extraction
< source >( text: str normalize: typing.Optional[bool] = None prompt_name: typing.Optional[str] = None truncate: typing.Optional[bool] = None truncation_direction: typing.Optional[typing.Literal['Left', 'Right']] = None model: typing.Optional[str] = None ) → np.ndarray
Parameters
- text (str) — The text to embed.
- model (str, optional) — The model to use for the conversational task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended conversational model will be used. Defaults to None.
- normalize (bool, optional) — Whether to normalize the embeddings or not. Only available on server powered by Text-Embedding-Inference.
- prompt_name (str, optional) —
The name of the prompt that should be used by for encoding. If not set, no prompt will be applied.
Must be a key in the Sentence Transformers configuration prompts dictionary.
For example if
prompt_nameis “query” and thepromptsis {“query”: “query: ”,…}, then the sentence “What is the capital of France?” will be encoded as “query: What is the capital of France?” because the prompt text will be prepended before any text to encode. - truncate (bool, optional) — Whether to truncate the embeddings or not. Only available on server powered by Text-Embedding-Inference.
- truncation_direction (Literal[“Left”, “Right”], optional) — Which side of the input should be truncated when truncate=True is passed.
Returns
np.ndarray
The embedding representing the input text as a float32 numpy array.
Raises
[InferenceTimeoutError] or HTTPError
- [InferenceTimeoutError] — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Generate embeddings for a given text.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.feature_extraction("Hi, who are you?")
array([[ 2.424802 , 2.93384 , 1.1750331 , ..., 1.240499, -0.13776633, -0.7889173 ],
[-0.42943227, -0.6364878 , -1.693462 , ..., 0.41978157, -2.4336355 , 0.6162071 ],
...,
[ 0.28552425, -0.928395 , -1.2077185 , ..., 0.76810825, -2.1069427 , 0.6236161 ]], dtype=float32)fill_mask
< source >( text: str model: typing.Optional[str] = None targets: typing.Optional[typing.List[str]] = None top_k: typing.Optional[int] = None ) → List[FillMaskOutputElement]
Parameters
- text (
str) — a string to be filled from, must contain the [MASK] token (check model card for exact name of the mask). - model (
str, optional) — The model to use for the fill mask task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended fill mask model will be used. - targets (
List[str, optional) — When passed, the model will limit the scores to the passed targets instead of looking up in the whole vocabulary. If the provided targets are not in the model vocab, they will be tokenized and the first resulting token will be used (with a warning, and that might be slower). - top_k (
int, optional) — When passed, overrides the number of predictions to return.
Returns
List[FillMaskOutputElement]
a list of FillMaskOutputElement items containing the predicted label, associated probability, token reference, and completed text.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Fill in a hole with a missing word (token to be precise).
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.fill_mask("The goal of life is <mask>.")
[
FillMaskOutputElement(score=0.06897063553333282, token=11098, token_str=' happiness', sequence='The goal of life is happiness.'),
FillMaskOutputElement(score=0.06554922461509705, token=45075, token_str=' immortality', sequence='The goal of life is immortality.')
]get_endpoint_info
< source >( model: typing.Optional[str] = None ) → Dict[str, Any]
Get information about the deployed endpoint.
This endpoint is only available on endpoints powered by Text-Generation-Inference (TGI) or Text-Embedding-Inference (TEI).
Endpoints powered by transformers return an empty payload.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient("meta-llama/Meta-Llama-3-70B-Instruct")
>>> client.get_endpoint_info()
{
'model_id': 'meta-llama/Meta-Llama-3-70B-Instruct',
'model_sha': None,
'model_dtype': 'torch.float16',
'model_device_type': 'cuda',
'model_pipeline_tag': None,
'max_concurrent_requests': 128,
'max_best_of': 2,
'max_stop_sequences': 4,
'max_input_length': 8191,
'max_total_tokens': 8192,
'waiting_served_ratio': 0.3,
'max_batch_total_tokens': 1259392,
'max_waiting_tokens': 20,
'max_batch_size': None,
'validation_workers': 32,
'max_client_batch_size': 4,
'version': '2.0.2',
'sha': 'dccab72549635c7eb5ddb17f43f0b7cdff07c214',
'docker_label': 'sha-dccab72'
}get_model_status
< source >( model: typing.Optional[str] = None ) → ModelStatus
Parameters
- model (
str, optional) — Identifier of the model for witch the status gonna be checked. If model is not provided, the model associated with this instance of InferenceClient will be used. Only InferenceAPI service can be checked so the identifier cannot be a URL.
Returns
ModelStatus
An instance of ModelStatus dataclass, containing information, about the state of the model: load, state, compute type and framework.
Get the status of a model hosted on the Inference API.
This endpoint is mostly useful when you already know which model you want to use and want to check its availability. If you want to discover already deployed models, you should rather use list_deployed_models().
get_recommended_model
< source >( task: str ) → str
Parameters
- task (
str) — The Hugging Face task to get which model Hugging Face recommends. All available tasks can be found here.
Returns
str
Name of the model recommended for the input task.
Raises
ValueError
ValueError— If Hugging Face has no recommendation for the input task.
Get the model Hugging Face recommends for the input task.
health_check
< source >( model: typing.Optional[str] = None ) → bool
Check the health of the deployed endpoint.
Health check is only available with Inference Endpoints powered by Text-Generation-Inference (TGI) or Text-Embedding-Inference (TEI). For Inference API, please use InferenceClient.get_model_status() instead.
image_classification
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None function_to_apply: typing.Optional[ForwardRef('ImageClassificationOutputTransform')] = None top_k: typing.Optional[int] = None ) → List[ImageClassificationOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The image to classify. It can be raw bytes, an image file, or a URL to an online image. - model (
str, optional) — The model to use for image classification. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for image classification will be used. - function_to_apply (
"ImageClassificationOutputTransform", optional) — The function to apply to the output. - top_k (
int, optional) — When specified, limits the output to the top K most probable classes.
Returns
List[ImageClassificationOutputElement]
a list of ImageClassificationOutputElement items containing the predicted label and associated probability.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Perform image classification on the given image using the specified model.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.image_classification("https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Cute_dog.jpg/320px-Cute_dog.jpg")
[ImageClassificationOutputElement(label='Blenheim spaniel', score=0.9779096841812134), ...]image_segmentation
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None mask_threshold: typing.Optional[float] = None overlap_mask_area_threshold: typing.Optional[float] = None subtask: typing.Optional[ForwardRef('ImageSegmentationSubtask')] = None threshold: typing.Optional[float] = None ) → List[ImageSegmentationOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The image to segment. It can be raw bytes, an image file, or a URL to an online image. - model (
str, optional) — The model to use for image segmentation. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for image segmentation will be used. - mask_threshold (
float, optional) — Threshold to use when turning the predicted masks into binary values. - overlap_mask_area_threshold (
float, optional) — Mask overlap threshold to eliminate small, disconnected segments. - subtask (
"ImageSegmentationSubtask", optional) — Segmentation task to be performed, depending on model capabilities. - threshold (
float, optional) — Probability threshold to filter out predicted masks.
Returns
List[ImageSegmentationOutputElement]
A list of ImageSegmentationOutputElement items containing the segmented masks and associated attributes.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Perform image segmentation on the given image using the specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
image_to_image
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] prompt: typing.Optional[str] = None negative_prompt: typing.Optional[str] = None height: typing.Optional[int] = None width: typing.Optional[int] = None num_inference_steps: typing.Optional[int] = None guidance_scale: typing.Optional[float] = None model: typing.Optional[str] = None **kwargs ) → Image
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image for translation. It can be raw bytes, an image file, or a URL to an online image. - prompt (
str, optional) — The text prompt to guide the image generation. - negative_prompt (
str, optional) — A negative prompt to guide the translation process. - height (
int, optional) — The height in pixels of the generated image. - width (
int, optional) — The width in pixels of the generated image. - num_inference_steps (
int, optional) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. - guidance_scale (
float, optional) — Higher guidance scale encourages to generate images that are closely linked to the textprompt, usually at the expense of lower image quality. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. Defaults to None.
Returns
Image
The translated image.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Perform image-to-image translation using a specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
image_to_text
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None ) → ImageToTextOutput
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image to caption. It can be raw bytes, an image file, or a URL to an online image.. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. Defaults to None.
Returns
The generated text.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Takes an input image and return text.
Models can have very different outputs depending on your use case (image captioning, optical character recognition (OCR), Pix2Struct, etc). Please have a look to the model card to learn more about a model’s specificities.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.image_to_text("cat.jpg")
'a cat standing in a grassy field '
>>> client.image_to_text("https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Cute_dog.jpg/320px-Cute_dog.jpg")
'a dog laying on the grass next to a flower pot 'list_deployed_models
< source >( frameworks: typing.Union[NoneType, str, typing.Literal['all'], typing.List[str]] = None ) → Dict[str, List[str]]
Parameters
- frameworks (
Literal["all"]orList[str]orstr, optional) — The frameworks to filter on. By default only a subset of the available frameworks are tested. If set to “all”, all available frameworks will be tested. It is also possible to provide a single framework or a custom set of frameworks to check.
Returns
Dict[str, List[str]]
A dictionary mapping task names to a sorted list of model IDs.
List models deployed on the Serverless Inference API service.
This helper checks deployed models framework by framework. By default, it will check the 4 main frameworks that
are supported and account for 95% of the hosted models. However, if you want a complete list of models you can
specify frameworks="all" as input. Alternatively, if you know before-hand which framework you are interested
in, you can also restrict to search to this one (e.g. frameworks="text-generation-inference"). The more
frameworks are checked, the more time it will take.
This endpoint method does not return a live list of all models available for the Serverless Inference API service. It searches over a cached list of models that were recently available and the list may not be up to date. If you want to know the live status of a specific model, use get_model_status().
This endpoint method is mostly useful for discoverability. If you already know which model you want to use and want to check its availability, you can directly use get_model_status().
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
# Discover zero-shot-classification models currently deployed
>>> models = client.list_deployed_models()
>>> models["zero-shot-classification"]
['Narsil/deberta-large-mnli-zero-cls', 'facebook/bart-large-mnli', ...]
# List from only 1 framework
>>> client.list_deployed_models("text-generation-inference")
{'text-generation': ['bigcode/starcoder', 'meta-llama/Llama-2-70b-chat-hf', ...], ...}object_detection
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None threshold: typing.Optional[float] = None ) → List[ObjectDetectionOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The image to detect objects on. It can be raw bytes, an image file, or a URL to an online image. - model (
str, optional) — The model to use for object detection. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for object detection (DETR) will be used. - threshold (
float, optional) — The probability necessary to make a prediction.
Returns
List[ObjectDetectionOutputElement]
A list of ObjectDetectionOutputElement items containing the bounding boxes and associated attributes.
Raises
InferenceTimeoutError or HTTPError or ValueError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.ValueError— If the request output is not a List.
Perform object detection on the given image using the specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
post
< source >( json: typing.Union[str, typing.Dict, typing.List, NoneType] = None data: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path, NoneType] = None model: typing.Optional[str] = None task: typing.Optional[str] = None stream: bool = False ) → bytes
Parameters
- json (
Union[str, Dict, List], optional) — The JSON data to send in the request body, specific to each task. Defaults to None. - data (
Union[str, Path, bytes, BinaryIO], optional) — The content to send in the request body, specific to each task. It can be raw bytes, a pointer to an opened file, a local file path, or a URL to an online resource (image, audio file,…). If bothjsonanddataare passed,datawill take precedence. At leastjsonordatamust be provided. Defaults to None. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. Will override the model defined at the instance level. Defaults to None. - task (
str, optional) — The task to perform on the inference. All available tasks can be found here. Used only to default to a recommended model ifmodelis not provided. At leastmodelortaskmust be provided. Defaults to None. - stream (
bool, optional) — Whether to iterate over streaming APIs.
Returns
bytes
The raw bytes returned by the server.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Make a POST request to the inference server.
question_answering
< source >( question: str context: str model: typing.Optional[str] = None align_to_words: typing.Optional[bool] = None doc_stride: typing.Optional[int] = None handle_impossible_answer: typing.Optional[bool] = None max_answer_len: typing.Optional[int] = None max_question_len: typing.Optional[int] = None max_seq_len: typing.Optional[int] = None top_k: typing.Optional[int] = None ) → Union[QuestionAnsweringOutputElement, ListQuestionAnsweringOutputElement]
Parameters
- question (
str) — Question to be answered. - context (
str) — The context of the question. - model (
str) — The model to use for the question answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. - align_to_words (
bool, optional) — Attempts to align the answer to real words. Improves quality on space separated languages. Might hurt on non-space-separated languages (like Japanese or Chinese) - doc_stride (
int, optional) — If the context is too long to fit with the question for the model, it will be split in several chunks with some overlap. This argument controls the size of that overlap. - handle_impossible_answer (
bool, optional) — Whether to accept impossible as an answer. - max_answer_len (
int, optional) — The maximum length of predicted answers (e.g., only answers with a shorter length are considered). - max_question_len (
int, optional) — The maximum length of the question after tokenization. It will be truncated if needed. - max_seq_len (
int, optional) — The maximum length of the total sentence (context + question) in tokens of each chunk passed to the model. The context will be split in several chunks (using docStride as overlap) if needed. - top_k (
int, optional) — The number of answers to return (will be chosen by order of likelihood). Note that we return less than topk answers if there are not enough options available within the context.
Returns
Union[QuestionAnsweringOutputElement, ListQuestionAnsweringOutputElement]
When top_k is 1 or not provided, it returns a single QuestionAnsweringOutputElement.
When top_k is greater than 1, it returns a list of QuestionAnsweringOutputElement.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Retrieve the answer to a question from a given text.
sentence_similarity
< source >( sentence: str other_sentences: typing.List[str] model: typing.Optional[str] = None ) → List[float]
Parameters
- sentence (
str) — The main sentence to compare to others. - other_sentences (
List[str]) — The list of sentences to compare to. - model (
str, optional) — The model to use for the conversational task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended conversational model will be used. Defaults to None.
Returns
List[float]
The embedding representing the input text.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Compute the semantic similarity between a sentence and a list of other sentences by comparing their embeddings.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.sentence_similarity(
... "Machine learning is so easy.",
... other_sentences=[
... "Deep learning is so straightforward.",
... "This is so difficult, like rocket science.",
... "I can't believe how much I struggled with this.",
... ],
... )
[0.7785726189613342, 0.45876261591911316, 0.2906220555305481]summarization
< source >( text: str parameters: typing.Optional[typing.Dict[str, typing.Any]] = None model: typing.Optional[str] = None clean_up_tokenization_spaces: typing.Optional[bool] = None generate_parameters: typing.Optional[typing.Dict[str, typing.Any]] = None truncation: typing.Optional[ForwardRef('SummarizationTruncationStrategy')] = None ) → SummarizationOutput
Parameters
- text (
str) — The input text to summarize. - parameters (
Dict[str, Any], optional) — Additional parameters for summarization. Check out this page for more details. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for summarization will be used. - clean_up_tokenization_spaces (
bool, optional) — Whether to clean up the potential extra spaces in the text output. - generate_parameters (
Dict[str, Any], optional) — Additional parametrization of the text generation algorithm. - truncation (
"SummarizationTruncationStrategy", optional) — The truncation strategy to use.
Returns
The generated summary text.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Generate a summary of a given text using a specified model.
table_question_answering
< source >( table: typing.Dict[str, typing.Any] query: str model: typing.Optional[str] = None parameters: typing.Optional[typing.Dict[str, typing.Any]] = None ) → TableQuestionAnsweringOutputElement
Parameters
- table (
str) — A table of data represented as a dict of lists where entries are headers and the lists are all the values, all lists must have the same size. - query (
str) — The query in plain text that you want to ask the table. - model (
str) — The model to use for the table-question-answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. - parameters (
Dict[str, Any], optional) — Additional inference parameters. Defaults to None.
a table question answering output containing the answer, coordinates, cells and the aggregator used.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Retrieve the answer to a question from information given in a table.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> query = "How many stars does the transformers repository have?"
>>> table = {"Repository": ["Transformers", "Datasets", "Tokenizers"], "Stars": ["36542", "4512", "3934"]}
>>> client.table_question_answering(table, query, model="google/tapas-base-finetuned-wtq")
TableQuestionAnsweringOutputElement(answer='36542', coordinates=[[0, 1]], cells=['36542'], aggregator='AVERAGE')tabular_classification
< source >( table: typing.Dict[str, typing.Any] model: typing.Optional[str] = None ) → List
Parameters
- table (
Dict[str, Any]) — Set of attributes to classify. - model (
str, optional) — The model to use for the tabular classification task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended tabular classification model will be used. Defaults to None.
Returns
List
a list of labels, one per row in the initial table.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Classifying a target category (a group) based on a set of attributes.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> table = {
... "fixed_acidity": ["7.4", "7.8", "10.3"],
... "volatile_acidity": ["0.7", "0.88", "0.32"],
... "citric_acid": ["0", "0", "0.45"],
... "residual_sugar": ["1.9", "2.6", "6.4"],
... "chlorides": ["0.076", "0.098", "0.073"],
... "free_sulfur_dioxide": ["11", "25", "5"],
... "total_sulfur_dioxide": ["34", "67", "13"],
... "density": ["0.9978", "0.9968", "0.9976"],
... "pH": ["3.51", "3.2", "3.23"],
... "sulphates": ["0.56", "0.68", "0.82"],
... "alcohol": ["9.4", "9.8", "12.6"],
... }
>>> client.tabular_classification(table=table, model="julien-c/wine-quality")
["5", "5", "5"]tabular_regression
< source >( table: typing.Dict[str, typing.Any] model: typing.Optional[str] = None ) → List
Parameters
- table (
Dict[str, Any]) — Set of attributes stored in a table. The attributes used to predict the target can be both numerical and categorical. - model (
str, optional) — The model to use for the tabular regression task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended tabular regression model will be used. Defaults to None.
Returns
List
a list of predicted numerical target values.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Predicting a numerical target value given a set of attributes/features in a table.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> table = {
... "Height": ["11.52", "12.48", "12.3778"],
... "Length1": ["23.2", "24", "23.9"],
... "Length2": ["25.4", "26.3", "26.5"],
... "Length3": ["30", "31.2", "31.1"],
... "Species": ["Bream", "Bream", "Bream"],
... "Width": ["4.02", "4.3056", "4.6961"],
... }
>>> client.tabular_regression(table, model="scikit-learn/Fish-Weight")
[110, 120, 130]text_classification
< source >( text: str model: typing.Optional[str] = None top_k: typing.Optional[int] = None function_to_apply: typing.Optional[ForwardRef('TextClassificationOutputTransform')] = None ) → List[TextClassificationOutputElement]
Parameters
- text (
str) — A string to be classified. - model (
str, optional) — The model to use for the text classification task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended text classification model will be used. Defaults to None. - top_k (
int, optional) — When specified, limits the output to the top K most probable classes. - function_to_apply (
"TextClassificationOutputTransform", optional) — The function to apply to the output.
Returns
List[TextClassificationOutputElement]
a list of TextClassificationOutputElement items containing the predicted label and associated probability.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Perform text classification (e.g. sentiment-analysis) on the given text.
text_generation
< source >( prompt: str details: bool = False stream: bool = False model: typing.Optional[str] = None adapter_id: typing.Optional[str] = None best_of: typing.Optional[int] = None decoder_input_details: typing.Optional[bool] = None do_sample: typing.Optional[bool] = False frequency_penalty: typing.Optional[float] = None grammar: typing.Optional[huggingface_hub.inference._generated.types.text_generation.TextGenerationInputGrammarType] = None max_new_tokens: typing.Optional[int] = None repetition_penalty: typing.Optional[float] = None return_full_text: typing.Optional[bool] = False seed: typing.Optional[int] = None stop: typing.Optional[typing.List[str]] = None stop_sequences: typing.Optional[typing.List[str]] = None temperature: typing.Optional[float] = None top_k: typing.Optional[int] = None top_n_tokens: typing.Optional[int] = None top_p: typing.Optional[float] = None truncate: typing.Optional[int] = None typical_p: typing.Optional[float] = None watermark: typing.Optional[bool] = None ) → Union[str, TextGenerationOutput, Iterable[str], Iterable[TextGenerationStreamOutput]]
Parameters
- prompt (
str) — Input text. - details (
bool, optional) — By default, text_generation returns a string. Passdetails=Trueif you want a detailed output (tokens, probabilities, seed, finish reason, etc.). Only available for models running on with thetext-generation-inferencebackend. - stream (
bool, optional) — By default, text_generation returns the full generated text. Passstream=Trueif you want a stream of tokens to be returned. Only available for models running on with thetext-generation-inferencebackend. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. Defaults to None. - adapter_id (
str, optional) — Lora adapter id. - best_of (
int, optional) — Generate best_of sequences and return the one if the highest token logprobs. - decoder_input_details (
bool, optional) — Return the decoder input token logprobs and ids. You must setdetails=Trueas well for it to be taken into account. Defaults toFalse. - do_sample (
bool, optional) — Activate logits sampling - frequency_penalty (
float, optional) — Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. - grammar (TextGenerationInputGrammarType, optional) — Grammar constraints. Can be either a JSONSchema or a regex.
- max_new_tokens (
int, optional) — Maximum number of generated tokens. Defaults to 100. - repetition_penalty (
float, optional) — The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details. - return_full_text (
bool, optional) — Whether to prepend the prompt to the generated text - seed (
int, optional) — Random sampling seed - stop (
List[str], optional) — Stop generating tokens if a member ofstopis generated. - stop_sequences (
List[str], optional) — Deprecated argument. Usestopinstead. - temperature (
float, optional) — The value used to module the logits distribution. - top_n_tokens (
int, optional) — Return information about thetop_n_tokensmost likely tokens at each generation step, instead of just the sampled token. - top_k (
int, *optional`) — The number of highest probability vocabulary tokens to keep for top-k-filtering. - top_p (
float, *optional) -- If set to < 1, only the smallest set of most probable tokens with probabilities that add up totop_p` or higher are kept for generation. - truncate (
int, *optional`) — Truncate inputs tokens to the given size. - typical_p (
float, *optional`) — Typical Decoding mass See Typical Decoding for Natural Language Generation for more information - watermark (
bool, *optional`) — Watermarking with A Watermark for Large Language Models
Returns
Union[str, TextGenerationOutput, Iterable[str], Iterable[TextGenerationStreamOutput]]
Generated text returned from the server:
- if
stream=Falseanddetails=False, the generated text is returned as astr(default) - if
stream=Trueanddetails=False, the generated text is returned token by token as aIterable[str] - if
stream=Falseanddetails=True, the generated text is returned with more details as a TextGenerationOutput - if
details=Trueandstream=True, the generated text is returned token by token as a iterable of TextGenerationStreamOutput
Raises
ValidationError or InferenceTimeoutError or HTTPError
ValidationError— If input values are not valid. No HTTP call is made to the server.- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Given a prompt, generate the following text.
API endpoint is supposed to run with the text-generation-inference backend (TGI). This backend is the
go-to solution to run large language models at scale. However, for some smaller models (e.g. “gpt2”) the
default transformers + api-inference solution is still in use. Both approaches have very similar APIs, but
not exactly the same. This method is compatible with both approaches but some parameters are only available for
text-generation-inference. If some parameters are ignored, a warning message is triggered but the process
continues correctly.
To learn more about the TGI project, please refer to https://github.com/huggingface/text-generation-inference.
If you want to generate a response from chat messages, you should use the InferenceClient.chat_completion() method. It accepts a list of messages instead of a single text prompt and handles the chat templating for you.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
# Case 1: generate text
>>> client.text_generation("The huggingface_hub library is ", max_new_tokens=12)
'100% open source and built to be easy to use.'
# Case 2: iterate over the generated tokens. Useful for large generation.
>>> for token in client.text_generation("The huggingface_hub library is ", max_new_tokens=12, stream=True):
... print(token)
100
%
open
source
and
built
to
be
easy
to
use
.
# Case 3: get more details about the generation process.
>>> client.text_generation("The huggingface_hub library is ", max_new_tokens=12, details=True)
TextGenerationOutput(
generated_text='100% open source and built to be easy to use.',
details=TextGenerationDetails(
finish_reason='length',
generated_tokens=12,
seed=None,
prefill=[
TextGenerationPrefillOutputToken(id=487, text='The', logprob=None),
TextGenerationPrefillOutputToken(id=53789, text=' hugging', logprob=-13.171875),
(...)
TextGenerationPrefillOutputToken(id=204, text=' ', logprob=-7.0390625)
],
tokens=[
TokenElement(id=1425, text='100', logprob=-1.0175781, special=False),
TokenElement(id=16, text='%', logprob=-0.0463562, special=False),
(...)
TokenElement(id=25, text='.', logprob=-0.5703125, special=False)
],
best_of_sequences=None
)
)
# Case 4: iterate over the generated tokens with more details.
# Last object is more complete, containing the full generated text and the finish reason.
>>> for details in client.text_generation("The huggingface_hub library is ", max_new_tokens=12, details=True, stream=True):
... print(details)
...
TextGenerationStreamOutput(token=TokenElement(id=1425, text='100', logprob=-1.0175781, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=16, text='%', logprob=-0.0463562, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=1314, text=' open', logprob=-1.3359375, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=3178, text=' source', logprob=-0.28100586, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=273, text=' and', logprob=-0.5961914, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=3426, text=' built', logprob=-1.9423828, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=271, text=' to', logprob=-1.4121094, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=314, text=' be', logprob=-1.5224609, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=1833, text=' easy', logprob=-2.1132812, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=271, text=' to', logprob=-0.08520508, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=745, text=' use', logprob=-0.39453125, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(
id=25,
text='.',
logprob=-0.5703125,
special=False),
generated_text='100% open source and built to be easy to use.',
details=TextGenerationStreamOutputStreamDetails(finish_reason='length', generated_tokens=12, seed=None)
)
# Case 5: generate constrained output using grammar
>>> response = client.text_generation(
... prompt="I saw a puppy a cat and a raccoon during my bike ride in the park",
... model="HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1",
... max_new_tokens=100,
... repetition_penalty=1.3,
... grammar={
... "type": "json",
... "value": {
... "properties": {
... "location": {"type": "string"},
... "activity": {"type": "string"},
... "animals_seen": {"type": "integer", "minimum": 1, "maximum": 5},
... "animals": {"type": "array", "items": {"type": "string"}},
... },
... "required": ["location", "activity", "animals_seen", "animals"],
... },
... },
... )
>>> json.loads(response)
{
"activity": "bike riding",
"animals": ["puppy", "cat", "raccoon"],
"animals_seen": 3,
"location": "park"
}text_to_image
< source >( prompt: str negative_prompt: typing.Optional[typing.List[str]] = None height: typing.Optional[float] = None width: typing.Optional[float] = None num_inference_steps: typing.Optional[int] = None guidance_scale: typing.Optional[float] = None model: typing.Optional[str] = None scheduler: typing.Optional[str] = None target_size: typing.Optional[huggingface_hub.inference._generated.types.text_to_image.TextToImageTargetSize] = None seed: typing.Optional[int] = None **kwargs ) → Image
Parameters
- prompt (
str) — The prompt to generate an image from. - negative_prompt (
List[str, optional) — One or several prompt to guide what NOT to include in image generation. - height (
float, optional) — The height in pixels of the image to generate. - width (
float, optional) — The width in pixels of the image to generate. - num_inference_steps (
int, optional) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. - guidance_scale (
float, optional) — A higher guidance scale value encourages the model to generate images closely linked to the text prompt, but values too high may cause saturation and other artifacts. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended text-to-image model will be used. Defaults to None. - scheduler (
str, optional) — Override the scheduler with a compatible one. - target_size (
TextToImageTargetSize, optional) — The size in pixel of the output image - seed (
int, optional) — Seed for the random number generator.
Returns
Image
The generated image.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Generate an image based on a given text using a specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> image = client.text_to_image("An astronaut riding a horse on the moon.")
>>> image.save("astronaut.png")
>>> image = client.text_to_image(
... "An astronaut riding a horse on the moon.",
... negative_prompt="low resolution, blurry",
... model="stabilityai/stable-diffusion-2-1",
... )
>>> image.save("better_astronaut.png")text_to_speech
< source >( text: str model: typing.Optional[str] = None do_sample: typing.Optional[bool] = None early_stopping: typing.Union[bool, ForwardRef('TextToSpeechEarlyStoppingEnum'), NoneType] = None epsilon_cutoff: typing.Optional[float] = None eta_cutoff: typing.Optional[float] = None max_length: typing.Optional[int] = None max_new_tokens: typing.Optional[int] = None min_length: typing.Optional[int] = None min_new_tokens: typing.Optional[int] = None num_beam_groups: typing.Optional[int] = None num_beams: typing.Optional[int] = None penalty_alpha: typing.Optional[float] = None temperature: typing.Optional[float] = None top_k: typing.Optional[int] = None top_p: typing.Optional[float] = None typical_p: typing.Optional[float] = None use_cache: typing.Optional[bool] = None ) → bytes
Parameters
- text (
str) — The text to synthesize. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended text-to-speech model will be used. Defaults to None. - do_sample (
bool, optional) — Whether to use sampling instead of greedy decoding when generating new tokens. - early_stopping (
Union[bool, "TextToSpeechEarlyStoppingEnum", optional) — Controls the stopping condition for beam-based methods. - epsilon_cutoff (
float, optional) — If set to float strictly between 0 and 1, only tokens with a conditional probability greater than epsilon_cutoff will be sampled. In the paper, suggested values range from 3e-4 to 9e-4, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details. - eta_cutoff (
float, optional) — Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than either eta_cutoff or sqrt(eta_cutoff)- exp(-entropy(softmax(next_token_logits))). The latter term is intuitively the expected next token probability, scaled by sqrt(eta_cutoff). In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details. float strictly between 0 and 1, a token is only considered if it is greater than either
- eta_cutoff (
float, optional) — Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than either eta_cutoff or sqrt(eta_cutoff)- exp(-entropy(softmax(next_token_logits))). The latter term is intuitively the expected next token probability, scaled by sqrt(eta_cutoff). In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details.
- max_length (
int, optional) — The maximum length (in tokens) of the generated text, including the input. - max_new_tokens (
int, optional) — The maximum number of tokens to generate. Takes precedence over maxLength. - min_length (
int, optional) — The minimum length (in tokens) of the generated text, including the input. - min_new_tokens (
int, optional) — The minimum number of tokens to generate. Takes precedence over maxLength. - num_beam_groups (
int, optional) — Number of groups to divide num_beams into in order to ensure diversity among different groups of beams. See this paper for more details. - num_beams (
int, optional) — Number of beams to use for beam search. - penalty_alpha (
float, optional) — The value balances the model confidence and the degeneration penalty in contrastive search decoding. - temperature (
float, optional) — The value used to modulate the next token probabilities. - top_k (
int, optional) — The number of highest probability vocabulary tokens to keep for top-k-filtering. - top_p (
float, optional) — If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. - typical_p (
float, optional) — Local typicality measures how similar the conditional probability of predicting a target token next is to the expected conditional probability of predicting a random token next, given the partial text already generated. If set to float < 1, the smallest set of the most locally typical tokens with probabilities that add up to typical_p or higher are kept for generation. See this paper for more details. - use_cache (
bool, optional) — Whether the model should use the past last key/values attentions to speed up decoding
Returns
bytes
The generated audio.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Synthesize an audio of a voice pronouncing a given text.
token_classification
< source >( text: str model: typing.Optional[str] = None aggregation_strategy: typing.Optional[ForwardRef('TokenClassificationAggregationStrategy')] = None ignore_labels: typing.Optional[typing.List[str]] = None stride: typing.Optional[int] = None ) → List[TokenClassificationOutputElement]
Parameters
- text (
str) — A string to be classified. - model (
str, optional) — The model to use for the token classification task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended token classification model will be used. Defaults to None. - aggregation_strategy (
"TokenClassificationAggregationStrategy", optional) — The strategy used to fuse tokens based on model predictions - ignore_labels (
List[str, optional) — A list of labels to ignore - stride (
int, optional) — The number of overlapping tokens between chunks when splitting the input text.
Returns
List[TokenClassificationOutputElement]
List of TokenClassificationOutputElement items containing the entity group, confidence score, word, start and end index.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Perform token classification on the given text. Usually used for sentence parsing, either grammatical, or Named Entity Recognition (NER) to understand keywords contained within text.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.token_classification("My name is Sarah Jessica Parker but you can call me Jessica")
[
TokenClassificationOutputElement(
entity_group='PER',
score=0.9971321225166321,
word='Sarah Jessica Parker',
start=11,
end=31,
),
TokenClassificationOutputElement(
entity_group='PER',
score=0.9773476123809814,
word='Jessica',
start=52,
end=59,
)
]translation
< source >( text: str model: typing.Optional[str] = None src_lang: typing.Optional[str] = None tgt_lang: typing.Optional[str] = None clean_up_tokenization_spaces: typing.Optional[bool] = None truncation: typing.Optional[ForwardRef('TranslationTruncationStrategy')] = None generate_parameters: typing.Optional[typing.Dict[str, typing.Any]] = None ) → TranslationOutput
Parameters
- text (
str) — A string to be translated. - model (
str, optional) — The model to use for the translation task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended translation model will be used. Defaults to None. - src_lang (
str, optional) — The source language of the text. Required for models that can translate from multiple languages. - tgt_lang (
str, optional) — Target language to translate to. Required for models that can translate to multiple languages. - clean_up_tokenization_spaces (
bool, optional) — Whether to clean up the potential extra spaces in the text output. - truncation (
"TranslationTruncationStrategy", optional) — The truncation strategy to use. - generate_parameters (
Dict[str, Any], optional) — Additional parametrization of the text generation algorithm.
Returns
The generated translated text.
Raises
InferenceTimeoutError or HTTPError or ValueError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.ValueError— If only one of thesrc_langandtgt_langarguments are provided.
Convert text from one language to another.
Check out https://huggingface.co/tasks/translation for more information on how to choose the best model for
your specific use case. Source and target languages usually depend on the model.
However, it is possible to specify source and target languages for certain models. If you are working with one of these models,
you can use src_lang and tgt_lang arguments to pass the relevant information.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.translation("My name is Wolfgang and I live in Berlin")
'Mein Name ist Wolfgang und ich lebe in Berlin.'
>>> client.translation("My name is Wolfgang and I live in Berlin", model="Helsinki-NLP/opus-mt-en-fr")
TranslationOutput(translation_text='Je m'appelle Wolfgang et je vis à Berlin.')visual_question_answering
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] question: str model: typing.Optional[str] = None top_k: typing.Optional[int] = None ) → List[VisualQuestionAnsweringOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image for the context. It can be raw bytes, an image file, or a URL to an online image. - question (
str) — Question to be answered. - model (
str, optional) — The model to use for the visual question answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended visual question answering model will be used. Defaults to None. - top_k (
int, optional) — The number of answers to return (will be chosen by order of likelihood). Note that we return less than topk answers if there are not enough options available within the context.
Returns
List[VisualQuestionAnsweringOutputElement]
a list of VisualQuestionAnsweringOutputElement items containing the predicted label and associated probability.
Raises
InferenceTimeoutError or HTTPError
InferenceTimeoutError— If the model is unavailable or the request times out.HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Answering open-ended questions based on an image.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.visual_question_answering(
... image="https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg",
... question="What is the animal doing?"
... )
[
VisualQuestionAnsweringOutputElement(score=0.778609573841095, answer='laying down'),
VisualQuestionAnsweringOutputElement(score=0.6957435607910156, answer='sitting'),
]zero_shot_classification
< source >( text: str candidate_labels: typing.List[str] = None multi_label: typing.Optional[bool] = False hypothesis_template: typing.Optional[str] = None model: typing.Optional[str] = None labels: typing.List[str] = None ) → List[ZeroShotClassificationOutputElement]
Parameters
- text (
str) — The input text to classify. - candidate_labels (
List[str]) — The set of possible class labels to classify the text into. - labels (
List[str], optional) — (deprecated) List of strings. Each string is the verbalization of a possible label for the input text. - multi_label (
bool, optional) — Whether multiple candidate labels can be true. If false, the scores are normalized such that the sum of the label likelihoods for each sequence is 1. If true, the labels are considered independent and probabilities are normalized for each candidate. - hypothesis_template (
str, optional) — The sentence used in conjunction with candidateLabels to attempt the text classification by replacing the placeholder with the candidate labels. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. If not provided, the default recommended zero-shot classification model will be used.
Returns
List[ZeroShotClassificationOutputElement]
List of ZeroShotClassificationOutputElement items containing the predicted labels and their confidence.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Provide as input a text and a set of candidate labels to classify the input text.
Example with multi_label=False:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> text = (
... "A new model offers an explanation for how the Galilean satellites formed around the solar system's"
... "largest world. Konstantin Batygin did not set out to solve one of the solar system's most puzzling"
... " mysteries when he went for a run up a hill in Nice, France."
... )
>>> labels = ["space & cosmos", "scientific discovery", "microbiology", "robots", "archeology"]
>>> client.zero_shot_classification(text, labels)
[
ZeroShotClassificationOutputElement(label='scientific discovery', score=0.7961668968200684),
ZeroShotClassificationOutputElement(label='space & cosmos', score=0.18570658564567566),
ZeroShotClassificationOutputElement(label='microbiology', score=0.00730885099619627),
ZeroShotClassificationOutputElement(label='archeology', score=0.006258360575884581),
ZeroShotClassificationOutputElement(label='robots', score=0.004559356719255447),
]
>>> client.zero_shot_classification(text, labels, multi_label=True)
[
ZeroShotClassificationOutputElement(label='scientific discovery', score=0.9829297661781311),
ZeroShotClassificationOutputElement(label='space & cosmos', score=0.755190908908844),
ZeroShotClassificationOutputElement(label='microbiology', score=0.0005462635890580714),
ZeroShotClassificationOutputElement(label='archeology', score=0.00047131875180639327),
ZeroShotClassificationOutputElement(label='robots', score=0.00030448526376858354),
]Example with multi_label=True and a custom hypothesis_template:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.zero_shot_classification(
... text="I really like our dinner and I'm very happy. I don't like the weather though.",
... labels=["positive", "negative", "pessimistic", "optimistic"],
... multi_label=True,
... hypothesis_template="This text is {} towards the weather"
... )
[
ZeroShotClassificationOutputElement(label='negative', score=0.9231801629066467),
ZeroShotClassificationOutputElement(label='pessimistic', score=0.8760990500450134),
ZeroShotClassificationOutputElement(label='optimistic', score=0.0008674879791215062),
ZeroShotClassificationOutputElement(label='positive', score=0.0005250611575320363)
]zero_shot_image_classification
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] candidate_labels: typing.Optional[typing.List[str]] = None model: typing.Optional[str] = None hypothesis_template: typing.Optional[str] = None labels: typing.Optional[typing.List[str]] = None ) → List[ZeroShotImageClassificationOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image to caption. It can be raw bytes, an image file, or a URL to an online image. - candidate_labels (
List[str]) — The candidate labels for this image - labels (
List[str], optional) — (deprecated) List of string possible labels. There must be at least 2 labels. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. If not provided, the default recommended zero-shot image classification model will be used. - hypothesis_template (
str, optional) — The sentence used in conjunction with candidateLabels to attempt the text classification by replacing the placeholder with the candidate labels.
Returns
List[ZeroShotImageClassificationOutputElement]
List of ZeroShotImageClassificationOutputElement items containing the predicted labels and their confidence.
Raises
InferenceTimeoutError or HTTPError
- InferenceTimeoutError — If the model is unavailable or the request times out.
HTTPError— If the request fails with an HTTP error status code other than HTTP 503.
Provide input image and text labels to predict text labels for the image.
Example:
>>> from huggingface_hub import InferenceClient
>>> client = InferenceClient()
>>> client.zero_shot_image_classification(
... "https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Cute_dog.jpg/320px-Cute_dog.jpg",
... labels=["dog", "cat", "horse"],
... )
[ZeroShotImageClassificationOutputElement(label='dog', score=0.956),...]Async Inference Client
An async version of the client is also provided, based on asyncio and aiohttp.
To use it, you can either install aiohttp directly or use the [inference] extra:
pip install --upgrade huggingface_hub[inference]
# or
# pip install aiohttpclass huggingface_hub.AsyncInferenceClient
< source >( model: typing.Optional[str] = None token: typing.Union[str, bool, NoneType] = None timeout: typing.Optional[float] = None headers: typing.Optional[typing.Dict[str, str]] = None cookies: typing.Optional[typing.Dict[str, str]] = None trust_env: bool = False proxies: typing.Optional[typing.Any] = None base_url: typing.Optional[str] = None api_key: typing.Optional[str] = None )
Parameters
- model (
str,optional) — The model to run inference with. Can be a model id hosted on the Hugging Face Hub, e.g.meta-llama/Meta-Llama-3-8B-Instructor a URL to a deployed Inference Endpoint. Defaults to None, in which case a recommended model is automatically selected for the task. Note: for better compatibility with OpenAI’s client,modelhas been aliased asbase_url. Those 2 arguments are mutually exclusive. If usingbase_urlfor chat completion, the/chat/completionssuffix path will be appended to the base URL (see the TGI Messages API documentation for details). When passing a URL asmodel, the client will not append any suffix path to it. - token (
strorbool, optional) — Hugging Face token. Will default to the locally saved token if not provided. Passtoken=Falseif you don’t want to send your token to the server. Note: for better compatibility with OpenAI’s client,tokenhas been aliased asapi_key. Those 2 arguments are mutually exclusive and have the exact same behavior. - timeout (
float,optional) — The maximum number of seconds to wait for a response from the server. Loading a new model in Inference API can take up to several minutes. Defaults to None, meaning it will loop until the server is available. - headers (
Dict[str, str],optional) — Additional headers to send to the server. By default only the authorization and user-agent headers are sent. Values in this dictionary will override the default values. - cookies (
Dict[str, str],optional) — Additional cookies to send to the server. - trust_env (‘bool’, ‘optional’) —
Trust environment settings for proxy configuration if the parameter is
True(Falseby default). - proxies (
Any,optional) — Proxies to use for the request. - base_url (
str,optional) — Base URL to run inference. This is a duplicated argument frommodelto make InferenceClient follow the same pattern asopenai.OpenAIclient. Cannot be used ifmodelis set. Defaults to None. - api_key (
str,optional) — Token to use for authentication. This is a duplicated argument fromtokento make InferenceClient follow the same pattern asopenai.OpenAIclient. Cannot be used iftokenis set. Defaults to None.
Initialize a new Inference Client.
InferenceClient aims to provide a unified experience to perform inference. The client can be used seamlessly with either the (free) Inference API or self-hosted Inference Endpoints.
audio_classification
< source >( audio: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None top_k: typing.Optional[int] = None function_to_apply: typing.Optional[ForwardRef('AudioClassificationOutputTransform')] = None ) → List[AudioClassificationOutputElement]
Parameters
- audio (Union[str, Path, bytes, BinaryIO]) — The audio content to classify. It can be raw audio bytes, a local audio file, or a URL pointing to an audio file.
- model (
str, optional) — The model to use for audio classification. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for audio classification will be used. - top_k (
int, optional) — When specified, limits the output to the top K most probable classes. - function_to_apply (
"AudioClassificationOutputTransform", optional) — The function to apply to the output.
Returns
List[AudioClassificationOutputElement]
List of AudioClassificationOutputElement items containing the predicted labels and their confidence.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Perform audio classification on the provided audio content.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.audio_classification("audio.flac")
[
AudioClassificationOutputElement(score=0.4976358711719513, label='hap'),
AudioClassificationOutputElement(score=0.3677836060523987, label='neu'),
...
]audio_to_audio
< source >( audio: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None ) → List[AudioToAudioOutputElement]
Parameters
- audio (Union[str, Path, bytes, BinaryIO]) — The audio content for the model. It can be raw audio bytes, a local audio file, or a URL pointing to an audio file.
- model (
str, optional) — The model can be any model which takes an audio file and returns another audio file. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for audio_to_audio will be used.
Returns
List[AudioToAudioOutputElement]
A list of AudioToAudioOutputElement items containing audios label, content-type, and audio content in blob.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
InferenceTimeoutError— If the model is unavailable or the request times out.aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Performs multiple tasks related to audio-to-audio depending on the model (eg: speech enhancement, source separation).
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> audio_output = await client.audio_to_audio("audio.flac")
>>> async for i, item in enumerate(audio_output):
>>> with open(f"output_{i}.flac", "wb") as f:
f.write(item.blob)automatic_speech_recognition
< source >( audio: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None ) → AutomaticSpeechRecognitionOutput
Parameters
- audio (Union[str, Path, bytes, BinaryIO]) — The content to transcribe. It can be raw audio bytes, local audio file, or a URL to an audio file.
- model (
str, optional) — The model to use for ASR. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for ASR will be used.
Returns
An item containing the transcribed text and optionally the timestamp chunks.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Perform automatic speech recognition (ASR or audio-to-text) on the given audio content.
chat_completion
< source >( messages: typing.List[typing.Dict] model: typing.Optional[str] = None stream: bool = False frequency_penalty: typing.Optional[float] = None logit_bias: typing.Optional[typing.List[float]] = None logprobs: typing.Optional[bool] = None max_tokens: typing.Optional[int] = None n: typing.Optional[int] = None presence_penalty: typing.Optional[float] = None response_format: typing.Optional[huggingface_hub.inference._generated.types.chat_completion.ChatCompletionInputGrammarType] = None seed: typing.Optional[int] = None stop: typing.Optional[typing.List[str]] = None stream_options: typing.Optional[huggingface_hub.inference._generated.types.chat_completion.ChatCompletionInputStreamOptions] = None temperature: typing.Optional[float] = None tool_choice: typing.Union[huggingface_hub.inference._generated.types.chat_completion.ChatCompletionInputToolType, str, NoneType] = None tool_prompt: typing.Optional[str] = None tools: typing.Optional[typing.List[huggingface_hub.inference._generated.types.chat_completion.ToolElement]] = None top_logprobs: typing.Optional[int] = None top_p: typing.Optional[float] = None ) → ChatCompletionOutput or Iterable of ChatCompletionStreamOutput
Parameters
- messages (List of ChatCompletionInputMessage) — Conversation history consisting of roles and content pairs.
- model (
str, optional) — The model to use for chat-completion. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for chat-based text-generation will be used. See https://huggingface.co/tasks/text-generation for more details.If
modelis a model ID, it is passed to the server as themodelparameter. If you want to define a custom URL while settingmodelin the request payload, you must setbase_urlwhen initializing InferenceClient. - frequency_penalty (
float, optional) — Penalizes new tokens based on their existing frequency in the text so far. Range: [-2.0, 2.0]. Defaults to 0.0. - logit_bias (
List[float], optional) — Modify the likelihood of specified tokens appearing in the completion. Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token. Defaults to None. - logprobs (
bool, optional) — Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. - max_tokens (
int, optional) — Maximum number of tokens allowed in the response. Defaults to 100. - n (
int, optional) — UNUSED. - presence_penalty (
float, optional) — Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics. - response_format (ChatCompletionInputGrammarType, optional) — Grammar constraints. Can be either a JSONSchema or a regex.
- seed (Optional
int, optional) — Seed for reproducible control flow. Defaults to None. - stop (Optional
str, optional) — Up to four strings which trigger the end of the response. Defaults to None. - stream (
bool, optional) — Enable realtime streaming of responses. Defaults to False. - stream_options (ChatCompletionInputStreamOptions, optional) — Options for streaming completions.
- temperature (
float, optional) — Controls randomness of the generations. Lower values ensure less random completions. Range: [0, 2]. Defaults to 1.0. - top_logprobs (
int, optional) — An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used. - top_p (
float, optional) — Fraction of the most likely next words to sample from. Must be between 0 and 1. Defaults to 1.0. - tool_choice (ChatCompletionInputToolType or
str, optional) — The tool to use for the completion. Defaults to “auto”. - tool_prompt (
str, optional) — A prompt to be appended before the tools. - tools (List of ToolElement, optional) — A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for.
Returns
ChatCompletionOutput or Iterable of ChatCompletionStreamOutput
Generated text returned from the server:
- if
stream=False, the generated text is returned as a ChatCompletionOutput (default). - if
stream=True, the generated text is returned token by token as a sequence of ChatCompletionStreamOutput.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
A method for completing conversations using a specified language model.
The client.chat_completion method is aliased as client.chat.completions.create for compatibility with OpenAI’s client.
Inputs and outputs are strictly the same and using either syntax will yield the same results.
Check out the Inference guide
for more details about OpenAI’s compatibility.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> messages = [{"role": "user", "content": "What is the capital of France?"}]
>>> client = AsyncInferenceClient("meta-llama/Meta-Llama-3-8B-Instruct")
>>> await client.chat_completion(messages, max_tokens=100)
ChatCompletionOutput(
choices=[
ChatCompletionOutputComplete(
finish_reason='eos_token',
index=0,
message=ChatCompletionOutputMessage(
role='assistant',
content='The capital of France is Paris.',
name=None,
tool_calls=None
),
logprobs=None
)
],
created=1719907176,
id='',
model='meta-llama/Meta-Llama-3-8B-Instruct',
object='text_completion',
system_fingerprint='2.0.4-sha-f426a33',
usage=ChatCompletionOutputUsage(
completion_tokens=8,
prompt_tokens=17,
total_tokens=25
)
)Example using streaming:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> messages = [{"role": "user", "content": "What is the capital of France?"}]
>>> client = AsyncInferenceClient("meta-llama/Meta-Llama-3-8B-Instruct")
>>> async for token in await client.chat_completion(messages, max_tokens=10, stream=True):
... print(token)
ChatCompletionStreamOutput(choices=[ChatCompletionStreamOutputChoice(delta=ChatCompletionStreamOutputDelta(content='The', role='assistant'), index=0, finish_reason=None)], created=1710498504)
ChatCompletionStreamOutput(choices=[ChatCompletionStreamOutputChoice(delta=ChatCompletionStreamOutputDelta(content=' capital', role='assistant'), index=0, finish_reason=None)], created=1710498504)
(...)
ChatCompletionStreamOutput(choices=[ChatCompletionStreamOutputChoice(delta=ChatCompletionStreamOutputDelta(content=' may', role='assistant'), index=0, finish_reason=None)], created=1710498504)Example using OpenAI’s syntax:
# Must be run in an async context
# instead of `from openai import OpenAI`
from huggingface_hub import AsyncInferenceClient
# instead of `client = OpenAI(...)`
client = AsyncInferenceClient(
base_url=...,
api_key=...,
)
output = await client.chat.completions.create(
model="meta-llama/Meta-Llama-3-8B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Count to 10"},
],
stream=True,
max_tokens=1024,
)
for chunk in output:
print(chunk.choices[0].delta.content)Example using Image + Text as input:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
# provide a remote URL
>>> image_url ="https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
# or a base64-encoded image
>>> image_path = "/path/to/image.jpeg"
>>> with open(image_path, "rb") as f:
... base64_image = base64.b64encode(f.read()).decode("utf-8")
>>> image_url = f"data:image/jpeg;base64,{base64_image}"
>>> client = AsyncInferenceClient("meta-llama/Llama-3.2-11B-Vision-Instruct")
>>> output = await client.chat.completions.create(
... messages=[
... {
... "role": "user",
... "content": [
... {
... "type": "image_url",
... "image_url": {"url": image_url},
... },
... {
... "type": "text",
... "text": "Describe this image in one sentence.",
... },
... ],
... },
... ],
... )
>>> output
The image depicts the iconic Statue of Liberty situated in New York Harbor, New York, on a clear day.Example using tools:
# Must be run in an async context
>>> client = AsyncInferenceClient("meta-llama/Meta-Llama-3-70B-Instruct")
>>> messages = [
... {
... "role": "system",
... "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.",
... },
... {
... "role": "user",
... "content": "What's the weather like the next 3 days in San Francisco, CA?",
... },
... ]
>>> tools = [
... {
... "type": "function",
... "function": {
... "name": "get_current_weather",
... "description": "Get the current weather",
... "parameters": {
... "type": "object",
... "properties": {
... "location": {
... "type": "string",
... "description": "The city and state, e.g. San Francisco, CA",
... },
... "format": {
... "type": "string",
... "enum": ["celsius", "fahrenheit"],
... "description": "The temperature unit to use. Infer this from the users location.",
... },
... },
... "required": ["location", "format"],
... },
... },
... },
... {
... "type": "function",
... "function": {
... "name": "get_n_day_weather_forecast",
... "description": "Get an N-day weather forecast",
... "parameters": {
... "type": "object",
... "properties": {
... "location": {
... "type": "string",
... "description": "The city and state, e.g. San Francisco, CA",
... },
... "format": {
... "type": "string",
... "enum": ["celsius", "fahrenheit"],
... "description": "The temperature unit to use. Infer this from the users location.",
... },
... "num_days": {
... "type": "integer",
... "description": "The number of days to forecast",
... },
... },
... "required": ["location", "format", "num_days"],
... },
... },
... },
... ]
>>> response = await client.chat_completion(
... model="meta-llama/Meta-Llama-3-70B-Instruct",
... messages=messages,
... tools=tools,
... tool_choice="auto",
... max_tokens=500,
... )
>>> response.choices[0].message.tool_calls[0].function
ChatCompletionOutputFunctionDefinition(
arguments={
'location': 'San Francisco, CA',
'format': 'fahrenheit',
'num_days': 3
},
name='get_n_day_weather_forecast',
description=None
)Example using response_format:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient("meta-llama/Meta-Llama-3-70B-Instruct")
>>> messages = [
... {
... "role": "user",
... "content": "I saw a puppy a cat and a raccoon during my bike ride in the park. What did I saw and when?",
... },
... ]
>>> response_format = {
... "type": "json",
... "value": {
... "properties": {
... "location": {"type": "string"},
... "activity": {"type": "string"},
... "animals_seen": {"type": "integer", "minimum": 1, "maximum": 5},
... "animals": {"type": "array", "items": {"type": "string"}},
... },
... "required": ["location", "activity", "animals_seen", "animals"],
... },
... }
>>> response = await client.chat_completion(
... messages=messages,
... response_format=response_format,
... max_tokens=500,
)
>>> response.choices[0].message.content
'{
y": "bike ride",
": ["puppy", "cat", "raccoon"],
_seen": 3,
n": "park"}'Close all open sessions.
By default, ‘aiohttp.ClientSession’ objects are closed automatically when a call is completed. However, if you are streaming data from the server and you stop before the stream is complete, you must call this method to close the session properly.
Another possibility is to use an async context (e.g. async with AsyncInferenceClient(): ...).
document_question_answering
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] question: str model: typing.Optional[str] = None doc_stride: typing.Optional[int] = None handle_impossible_answer: typing.Optional[bool] = None lang: typing.Optional[str] = None max_answer_len: typing.Optional[int] = None max_question_len: typing.Optional[int] = None max_seq_len: typing.Optional[int] = None top_k: typing.Optional[int] = None word_boxes: typing.Optional[typing.List[typing.Union[typing.List[float], str]]] = None ) → List[DocumentQuestionAnsweringOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image for the context. It can be raw bytes, an image file, or a URL to an online image. - question (
str) — Question to be answered. - model (
str, optional) — The model to use for the document question answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended document question answering model will be used. Defaults to None. - doc_stride (
int, optional) — If the words in the document are too long to fit with the question for the model, it will be split in several chunks with some overlap. This argument controls the size of that overlap. - handle_impossible_answer (
bool, optional) — Whether to accept impossible as an answer - lang (
str, optional) — Language to use while running OCR. Defaults to english. - max_answer_len (
int, optional) — The maximum length of predicted answers (e.g., only answers with a shorter length are considered). - max_question_len (
int, optional) — The maximum length of the question after tokenization. It will be truncated if needed. - max_seq_len (
int, optional) — The maximum length of the total sentence (context + question) in tokens of each chunk passed to the model. The context will be split in several chunks (using doc_stride as overlap) if needed. - top_k (
int, optional) — The number of answers to return (will be chosen by order of likelihood). Can return less than top_k answers if there are not enough options available within the context. - word_boxes (
List[Union[List[float], str, optional) — A list of words and bounding boxes (normalized 0->1000). If provided, the inference will skip the OCR step and use the provided bounding boxes instead.
Returns
List[DocumentQuestionAnsweringOutputElement]
a list of DocumentQuestionAnsweringOutputElement items containing the predicted label, associated probability, word ids, and page number.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Answer questions on document images.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.document_question_answering(image="https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png", question="What is the invoice number?")
[DocumentQuestionAnsweringOutputElement(answer='us-001', end=16, score=0.9999666213989258, start=16, words=None)]feature_extraction
< source >( text: str normalize: typing.Optional[bool] = None prompt_name: typing.Optional[str] = None truncate: typing.Optional[bool] = None truncation_direction: typing.Optional[typing.Literal['Left', 'Right']] = None model: typing.Optional[str] = None ) → np.ndarray
Parameters
- text (str) — The text to embed.
- model (str, optional) — The model to use for the conversational task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended conversational model will be used. Defaults to None.
- normalize (bool, optional) — Whether to normalize the embeddings or not. Only available on server powered by Text-Embedding-Inference.
- prompt_name (str, optional) —
The name of the prompt that should be used by for encoding. If not set, no prompt will be applied.
Must be a key in the Sentence Transformers configuration prompts dictionary.
For example if
prompt_nameis “query” and thepromptsis {“query”: “query: ”,…}, then the sentence “What is the capital of France?” will be encoded as “query: What is the capital of France?” because the prompt text will be prepended before any text to encode. - truncate (bool, optional) — Whether to truncate the embeddings or not. Only available on server powered by Text-Embedding-Inference.
- truncation_direction (Literal[“Left”, “Right”], optional) — Which side of the input should be truncated when truncate=True is passed.
Returns
np.ndarray
The embedding representing the input text as a float32 numpy array.
Raises
[InferenceTimeoutError] or aiohttp.ClientResponseError
- [InferenceTimeoutError] — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Generate embeddings for a given text.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.feature_extraction("Hi, who are you?")
array([[ 2.424802 , 2.93384 , 1.1750331 , ..., 1.240499, -0.13776633, -0.7889173 ],
[-0.42943227, -0.6364878 , -1.693462 , ..., 0.41978157, -2.4336355 , 0.6162071 ],
...,
[ 0.28552425, -0.928395 , -1.2077185 , ..., 0.76810825, -2.1069427 , 0.6236161 ]], dtype=float32)fill_mask
< source >( text: str model: typing.Optional[str] = None targets: typing.Optional[typing.List[str]] = None top_k: typing.Optional[int] = None ) → List[FillMaskOutputElement]
Parameters
- text (
str) — a string to be filled from, must contain the [MASK] token (check model card for exact name of the mask). - model (
str, optional) — The model to use for the fill mask task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended fill mask model will be used. - targets (
List[str, optional) — When passed, the model will limit the scores to the passed targets instead of looking up in the whole vocabulary. If the provided targets are not in the model vocab, they will be tokenized and the first resulting token will be used (with a warning, and that might be slower). - top_k (
int, optional) — When passed, overrides the number of predictions to return.
Returns
List[FillMaskOutputElement]
a list of FillMaskOutputElement items containing the predicted label, associated probability, token reference, and completed text.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Fill in a hole with a missing word (token to be precise).
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.fill_mask("The goal of life is <mask>.")
[
FillMaskOutputElement(score=0.06897063553333282, token=11098, token_str=' happiness', sequence='The goal of life is happiness.'),
FillMaskOutputElement(score=0.06554922461509705, token=45075, token_str=' immortality', sequence='The goal of life is immortality.')
]get_endpoint_info
< source >( model: typing.Optional[str] = None ) → Dict[str, Any]
Get information about the deployed endpoint.
This endpoint is only available on endpoints powered by Text-Generation-Inference (TGI) or Text-Embedding-Inference (TEI).
Endpoints powered by transformers return an empty payload.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient("meta-llama/Meta-Llama-3-70B-Instruct")
>>> await client.get_endpoint_info()
{
'model_id': 'meta-llama/Meta-Llama-3-70B-Instruct',
'model_sha': None,
'model_dtype': 'torch.float16',
'model_device_type': 'cuda',
'model_pipeline_tag': None,
'max_concurrent_requests': 128,
'max_best_of': 2,
'max_stop_sequences': 4,
'max_input_length': 8191,
'max_total_tokens': 8192,
'waiting_served_ratio': 0.3,
'max_batch_total_tokens': 1259392,
'max_waiting_tokens': 20,
'max_batch_size': None,
'validation_workers': 32,
'max_client_batch_size': 4,
'version': '2.0.2',
'sha': 'dccab72549635c7eb5ddb17f43f0b7cdff07c214',
'docker_label': 'sha-dccab72'
}get_model_status
< source >( model: typing.Optional[str] = None ) → ModelStatus
Parameters
- model (
str, optional) — Identifier of the model for witch the status gonna be checked. If model is not provided, the model associated with this instance of InferenceClient will be used. Only InferenceAPI service can be checked so the identifier cannot be a URL.
Returns
ModelStatus
An instance of ModelStatus dataclass, containing information, about the state of the model: load, state, compute type and framework.
Get the status of a model hosted on the Inference API.
This endpoint is mostly useful when you already know which model you want to use and want to check its availability. If you want to discover already deployed models, you should rather use list_deployed_models().
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.get_model_status("meta-llama/Meta-Llama-3-8B-Instruct")
ModelStatus(loaded=True, state='Loaded', compute_type='gpu', framework='text-generation-inference')get_recommended_model
< source >( task: str ) → str
Parameters
- task (
str) — The Hugging Face task to get which model Hugging Face recommends. All available tasks can be found here.
Returns
str
Name of the model recommended for the input task.
Raises
ValueError
ValueError— If Hugging Face has no recommendation for the input task.
Get the model Hugging Face recommends for the input task.
health_check
< source >( model: typing.Optional[str] = None ) → bool
Check the health of the deployed endpoint.
Health check is only available with Inference Endpoints powered by Text-Generation-Inference (TGI) or Text-Embedding-Inference (TEI). For Inference API, please use InferenceClient.get_model_status() instead.
image_classification
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None function_to_apply: typing.Optional[ForwardRef('ImageClassificationOutputTransform')] = None top_k: typing.Optional[int] = None ) → List[ImageClassificationOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The image to classify. It can be raw bytes, an image file, or a URL to an online image. - model (
str, optional) — The model to use for image classification. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for image classification will be used. - function_to_apply (
"ImageClassificationOutputTransform", optional) — The function to apply to the output. - top_k (
int, optional) — When specified, limits the output to the top K most probable classes.
Returns
List[ImageClassificationOutputElement]
a list of ImageClassificationOutputElement items containing the predicted label and associated probability.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Perform image classification on the given image using the specified model.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.image_classification("https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Cute_dog.jpg/320px-Cute_dog.jpg")
[ImageClassificationOutputElement(label='Blenheim spaniel', score=0.9779096841812134), ...]image_segmentation
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None mask_threshold: typing.Optional[float] = None overlap_mask_area_threshold: typing.Optional[float] = None subtask: typing.Optional[ForwardRef('ImageSegmentationSubtask')] = None threshold: typing.Optional[float] = None ) → List[ImageSegmentationOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The image to segment. It can be raw bytes, an image file, or a URL to an online image. - model (
str, optional) — The model to use for image segmentation. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for image segmentation will be used. - mask_threshold (
float, optional) — Threshold to use when turning the predicted masks into binary values. - overlap_mask_area_threshold (
float, optional) — Mask overlap threshold to eliminate small, disconnected segments. - subtask (
"ImageSegmentationSubtask", optional) — Segmentation task to be performed, depending on model capabilities. - threshold (
float, optional) — Probability threshold to filter out predicted masks.
Returns
List[ImageSegmentationOutputElement]
A list of ImageSegmentationOutputElement items containing the segmented masks and associated attributes.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Perform image segmentation on the given image using the specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.image_segmentation("cat.jpg")
[ImageSegmentationOutputElement(score=0.989008, label='LABEL_184', mask=<PIL.PngImagePlugin.PngImageFile image mode=L size=400x300 at 0x7FDD2B129CC0>), ...]image_to_image
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] prompt: typing.Optional[str] = None negative_prompt: typing.Optional[str] = None height: typing.Optional[int] = None width: typing.Optional[int] = None num_inference_steps: typing.Optional[int] = None guidance_scale: typing.Optional[float] = None model: typing.Optional[str] = None **kwargs ) → Image
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image for translation. It can be raw bytes, an image file, or a URL to an online image. - prompt (
str, optional) — The text prompt to guide the image generation. - negative_prompt (
str, optional) — A negative prompt to guide the translation process. - height (
int, optional) — The height in pixels of the generated image. - width (
int, optional) — The width in pixels of the generated image. - num_inference_steps (
int, optional) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. - guidance_scale (
float, optional) — Higher guidance scale encourages to generate images that are closely linked to the textprompt, usually at the expense of lower image quality. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. Defaults to None.
Returns
Image
The translated image.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Perform image-to-image translation using a specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
image_to_text
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None ) → ImageToTextOutput
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image to caption. It can be raw bytes, an image file, or a URL to an online image.. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. Defaults to None.
Returns
The generated text.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Takes an input image and return text.
Models can have very different outputs depending on your use case (image captioning, optical character recognition (OCR), Pix2Struct, etc). Please have a look to the model card to learn more about a model’s specificities.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.image_to_text("cat.jpg")
'a cat standing in a grassy field '
>>> await client.image_to_text("https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Cute_dog.jpg/320px-Cute_dog.jpg")
'a dog laying on the grass next to a flower pot 'list_deployed_models
< source >( frameworks: typing.Union[NoneType, str, typing.Literal['all'], typing.List[str]] = None ) → Dict[str, List[str]]
Parameters
- frameworks (
Literal["all"]orList[str]orstr, optional) — The frameworks to filter on. By default only a subset of the available frameworks are tested. If set to “all”, all available frameworks will be tested. It is also possible to provide a single framework or a custom set of frameworks to check.
Returns
Dict[str, List[str]]
A dictionary mapping task names to a sorted list of model IDs.
List models deployed on the Serverless Inference API service.
This helper checks deployed models framework by framework. By default, it will check the 4 main frameworks that
are supported and account for 95% of the hosted models. However, if you want a complete list of models you can
specify frameworks="all" as input. Alternatively, if you know before-hand which framework you are interested
in, you can also restrict to search to this one (e.g. frameworks="text-generation-inference"). The more
frameworks are checked, the more time it will take.
This endpoint method does not return a live list of all models available for the Serverless Inference API service. It searches over a cached list of models that were recently available and the list may not be up to date. If you want to know the live status of a specific model, use get_model_status().
This endpoint method is mostly useful for discoverability. If you already know which model you want to use and want to check its availability, you can directly use get_model_status().
Example:
# Must be run in an async contextthon
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
# Discover zero-shot-classification models currently deployed
>>> models = await client.list_deployed_models()
>>> models["zero-shot-classification"]
['Narsil/deberta-large-mnli-zero-cls', 'facebook/bart-large-mnli', ...]
# List from only 1 framework
>>> await client.list_deployed_models("text-generation-inference")
{'text-generation': ['bigcode/starcoder', 'meta-llama/Llama-2-70b-chat-hf', ...], ...}object_detection
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] model: typing.Optional[str] = None threshold: typing.Optional[float] = None ) → List[ObjectDetectionOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The image to detect objects on. It can be raw bytes, an image file, or a URL to an online image. - model (
str, optional) — The model to use for object detection. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for object detection (DETR) will be used. - threshold (
float, optional) — The probability necessary to make a prediction.
Returns
List[ObjectDetectionOutputElement]
A list of ObjectDetectionOutputElement items containing the bounding boxes and associated attributes.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError or ValueError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.ValueError— If the request output is not a List.
Perform object detection on the given image using the specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.object_detection("people.jpg")
[ObjectDetectionOutputElement(score=0.9486683011054993, label='person', box=ObjectDetectionBoundingBox(xmin=59, ymin=39, xmax=420, ymax=510)), ...]post
< source >( json: typing.Union[str, typing.Dict, typing.List, NoneType] = None data: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path, NoneType] = None model: typing.Optional[str] = None task: typing.Optional[str] = None stream: bool = False ) → bytes
Parameters
- json (
Union[str, Dict, List], optional) — The JSON data to send in the request body, specific to each task. Defaults to None. - data (
Union[str, Path, bytes, BinaryIO], optional) — The content to send in the request body, specific to each task. It can be raw bytes, a pointer to an opened file, a local file path, or a URL to an online resource (image, audio file,…). If bothjsonanddataare passed,datawill take precedence. At leastjsonordatamust be provided. Defaults to None. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. Will override the model defined at the instance level. Defaults to None. - task (
str, optional) — The task to perform on the inference. All available tasks can be found here. Used only to default to a recommended model ifmodelis not provided. At leastmodelortaskmust be provided. Defaults to None. - stream (
bool, optional) — Whether to iterate over streaming APIs.
Returns
bytes
The raw bytes returned by the server.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Make a POST request to the inference server.
question_answering
< source >( question: str context: str model: typing.Optional[str] = None align_to_words: typing.Optional[bool] = None doc_stride: typing.Optional[int] = None handle_impossible_answer: typing.Optional[bool] = None max_answer_len: typing.Optional[int] = None max_question_len: typing.Optional[int] = None max_seq_len: typing.Optional[int] = None top_k: typing.Optional[int] = None ) → Union[QuestionAnsweringOutputElement, ListQuestionAnsweringOutputElement]
Parameters
- question (
str) — Question to be answered. - context (
str) — The context of the question. - model (
str) — The model to use for the question answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. - align_to_words (
bool, optional) — Attempts to align the answer to real words. Improves quality on space separated languages. Might hurt on non-space-separated languages (like Japanese or Chinese) - doc_stride (
int, optional) — If the context is too long to fit with the question for the model, it will be split in several chunks with some overlap. This argument controls the size of that overlap. - handle_impossible_answer (
bool, optional) — Whether to accept impossible as an answer. - max_answer_len (
int, optional) — The maximum length of predicted answers (e.g., only answers with a shorter length are considered). - max_question_len (
int, optional) — The maximum length of the question after tokenization. It will be truncated if needed. - max_seq_len (
int, optional) — The maximum length of the total sentence (context + question) in tokens of each chunk passed to the model. The context will be split in several chunks (using docStride as overlap) if needed. - top_k (
int, optional) — The number of answers to return (will be chosen by order of likelihood). Note that we return less than topk answers if there are not enough options available within the context.
Returns
Union[QuestionAnsweringOutputElement, ListQuestionAnsweringOutputElement]
When top_k is 1 or not provided, it returns a single QuestionAnsweringOutputElement.
When top_k is greater than 1, it returns a list of QuestionAnsweringOutputElement.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Retrieve the answer to a question from a given text.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.question_answering(question="What's my name?", context="My name is Clara and I live in Berkeley.")
QuestionAnsweringOutputElement(answer='Clara', end=16, score=0.9326565265655518, start=11)sentence_similarity
< source >( sentence: str other_sentences: typing.List[str] model: typing.Optional[str] = None ) → List[float]
Parameters
- sentence (
str) — The main sentence to compare to others. - other_sentences (
List[str]) — The list of sentences to compare to. - model (
str, optional) — The model to use for the conversational task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended conversational model will be used. Defaults to None.
Returns
List[float]
The embedding representing the input text.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Compute the semantic similarity between a sentence and a list of other sentences by comparing their embeddings.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.sentence_similarity(
... "Machine learning is so easy.",
... other_sentences=[
... "Deep learning is so straightforward.",
... "This is so difficult, like rocket science.",
... "I can't believe how much I struggled with this.",
... ],
... )
[0.7785726189613342, 0.45876261591911316, 0.2906220555305481]summarization
< source >( text: str parameters: typing.Optional[typing.Dict[str, typing.Any]] = None model: typing.Optional[str] = None clean_up_tokenization_spaces: typing.Optional[bool] = None generate_parameters: typing.Optional[typing.Dict[str, typing.Any]] = None truncation: typing.Optional[ForwardRef('SummarizationTruncationStrategy')] = None ) → SummarizationOutput
Parameters
- text (
str) — The input text to summarize. - parameters (
Dict[str, Any], optional) — Additional parameters for summarization. Check out this page for more details. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended model for summarization will be used. - clean_up_tokenization_spaces (
bool, optional) — Whether to clean up the potential extra spaces in the text output. - generate_parameters (
Dict[str, Any], optional) — Additional parametrization of the text generation algorithm. - truncation (
"SummarizationTruncationStrategy", optional) — The truncation strategy to use.
Returns
The generated summary text.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Generate a summary of a given text using a specified model.
table_question_answering
< source >( table: typing.Dict[str, typing.Any] query: str model: typing.Optional[str] = None parameters: typing.Optional[typing.Dict[str, typing.Any]] = None ) → TableQuestionAnsweringOutputElement
Parameters
- table (
str) — A table of data represented as a dict of lists where entries are headers and the lists are all the values, all lists must have the same size. - query (
str) — The query in plain text that you want to ask the table. - model (
str) — The model to use for the table-question-answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. - parameters (
Dict[str, Any], optional) — Additional inference parameters. Defaults to None.
a table question answering output containing the answer, coordinates, cells and the aggregator used.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Retrieve the answer to a question from information given in a table.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> query = "How many stars does the transformers repository have?"
>>> table = {"Repository": ["Transformers", "Datasets", "Tokenizers"], "Stars": ["36542", "4512", "3934"]}
>>> await client.table_question_answering(table, query, model="google/tapas-base-finetuned-wtq")
TableQuestionAnsweringOutputElement(answer='36542', coordinates=[[0, 1]], cells=['36542'], aggregator='AVERAGE')tabular_classification
< source >( table: typing.Dict[str, typing.Any] model: typing.Optional[str] = None ) → List
Parameters
- table (
Dict[str, Any]) — Set of attributes to classify. - model (
str, optional) — The model to use for the tabular classification task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended tabular classification model will be used. Defaults to None.
Returns
List
a list of labels, one per row in the initial table.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Classifying a target category (a group) based on a set of attributes.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> table = {
... "fixed_acidity": ["7.4", "7.8", "10.3"],
... "volatile_acidity": ["0.7", "0.88", "0.32"],
... "citric_acid": ["0", "0", "0.45"],
... "residual_sugar": ["1.9", "2.6", "6.4"],
... "chlorides": ["0.076", "0.098", "0.073"],
... "free_sulfur_dioxide": ["11", "25", "5"],
... "total_sulfur_dioxide": ["34", "67", "13"],
... "density": ["0.9978", "0.9968", "0.9976"],
... "pH": ["3.51", "3.2", "3.23"],
... "sulphates": ["0.56", "0.68", "0.82"],
... "alcohol": ["9.4", "9.8", "12.6"],
... }
>>> await client.tabular_classification(table=table, model="julien-c/wine-quality")
["5", "5", "5"]tabular_regression
< source >( table: typing.Dict[str, typing.Any] model: typing.Optional[str] = None ) → List
Parameters
- table (
Dict[str, Any]) — Set of attributes stored in a table. The attributes used to predict the target can be both numerical and categorical. - model (
str, optional) — The model to use for the tabular regression task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended tabular regression model will be used. Defaults to None.
Returns
List
a list of predicted numerical target values.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Predicting a numerical target value given a set of attributes/features in a table.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> table = {
... "Height": ["11.52", "12.48", "12.3778"],
... "Length1": ["23.2", "24", "23.9"],
... "Length2": ["25.4", "26.3", "26.5"],
... "Length3": ["30", "31.2", "31.1"],
... "Species": ["Bream", "Bream", "Bream"],
... "Width": ["4.02", "4.3056", "4.6961"],
... }
>>> await client.tabular_regression(table, model="scikit-learn/Fish-Weight")
[110, 120, 130]text_classification
< source >( text: str model: typing.Optional[str] = None top_k: typing.Optional[int] = None function_to_apply: typing.Optional[ForwardRef('TextClassificationOutputTransform')] = None ) → List[TextClassificationOutputElement]
Parameters
- text (
str) — A string to be classified. - model (
str, optional) — The model to use for the text classification task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended text classification model will be used. Defaults to None. - top_k (
int, optional) — When specified, limits the output to the top K most probable classes. - function_to_apply (
"TextClassificationOutputTransform", optional) — The function to apply to the output.
Returns
List[TextClassificationOutputElement]
a list of TextClassificationOutputElement items containing the predicted label and associated probability.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Perform text classification (e.g. sentiment-analysis) on the given text.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.text_classification("I like you")
[
TextClassificationOutputElement(label='POSITIVE', score=0.9998695850372314),
TextClassificationOutputElement(label='NEGATIVE', score=0.0001304351753788069),
]text_generation
< source >( prompt: str details: bool = False stream: bool = False model: typing.Optional[str] = None adapter_id: typing.Optional[str] = None best_of: typing.Optional[int] = None decoder_input_details: typing.Optional[bool] = None do_sample: typing.Optional[bool] = False frequency_penalty: typing.Optional[float] = None grammar: typing.Optional[huggingface_hub.inference._generated.types.text_generation.TextGenerationInputGrammarType] = None max_new_tokens: typing.Optional[int] = None repetition_penalty: typing.Optional[float] = None return_full_text: typing.Optional[bool] = False seed: typing.Optional[int] = None stop: typing.Optional[typing.List[str]] = None stop_sequences: typing.Optional[typing.List[str]] = None temperature: typing.Optional[float] = None top_k: typing.Optional[int] = None top_n_tokens: typing.Optional[int] = None top_p: typing.Optional[float] = None truncate: typing.Optional[int] = None typical_p: typing.Optional[float] = None watermark: typing.Optional[bool] = None ) → Union[str, TextGenerationOutput, Iterable[str], Iterable[TextGenerationStreamOutput]]
Parameters
- prompt (
str) — Input text. - details (
bool, optional) — By default, text_generation returns a string. Passdetails=Trueif you want a detailed output (tokens, probabilities, seed, finish reason, etc.). Only available for models running on with thetext-generation-inferencebackend. - stream (
bool, optional) — By default, text_generation returns the full generated text. Passstream=Trueif you want a stream of tokens to be returned. Only available for models running on with thetext-generation-inferencebackend. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. Defaults to None. - adapter_id (
str, optional) — Lora adapter id. - best_of (
int, optional) — Generate best_of sequences and return the one if the highest token logprobs. - decoder_input_details (
bool, optional) — Return the decoder input token logprobs and ids. You must setdetails=Trueas well for it to be taken into account. Defaults toFalse. - do_sample (
bool, optional) — Activate logits sampling - frequency_penalty (
float, optional) — Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. - grammar (TextGenerationInputGrammarType, optional) — Grammar constraints. Can be either a JSONSchema or a regex.
- max_new_tokens (
int, optional) — Maximum number of generated tokens. Defaults to 100. - repetition_penalty (
float, optional) — The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details. - return_full_text (
bool, optional) — Whether to prepend the prompt to the generated text - seed (
int, optional) — Random sampling seed - stop (
List[str], optional) — Stop generating tokens if a member ofstopis generated. - stop_sequences (
List[str], optional) — Deprecated argument. Usestopinstead. - temperature (
float, optional) — The value used to module the logits distribution. - top_n_tokens (
int, optional) — Return information about thetop_n_tokensmost likely tokens at each generation step, instead of just the sampled token. - top_k (
int, *optional`) — The number of highest probability vocabulary tokens to keep for top-k-filtering. - top_p (
float, *optional) -- If set to < 1, only the smallest set of most probable tokens with probabilities that add up totop_p` or higher are kept for generation. - truncate (
int, *optional`) — Truncate inputs tokens to the given size. - typical_p (
float, *optional`) — Typical Decoding mass See Typical Decoding for Natural Language Generation for more information - watermark (
bool, *optional`) — Watermarking with A Watermark for Large Language Models
Returns
Union[str, TextGenerationOutput, Iterable[str], Iterable[TextGenerationStreamOutput]]
Generated text returned from the server:
- if
stream=Falseanddetails=False, the generated text is returned as astr(default) - if
stream=Trueanddetails=False, the generated text is returned token by token as aIterable[str] - if
stream=Falseanddetails=True, the generated text is returned with more details as a TextGenerationOutput - if
details=Trueandstream=True, the generated text is returned token by token as a iterable of TextGenerationStreamOutput
Raises
ValidationError or InferenceTimeoutError or aiohttp.ClientResponseError
ValidationError— If input values are not valid. No HTTP call is made to the server.- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Given a prompt, generate the following text.
API endpoint is supposed to run with the text-generation-inference backend (TGI). This backend is the
go-to solution to run large language models at scale. However, for some smaller models (e.g. “gpt2”) the
default transformers + api-inference solution is still in use. Both approaches have very similar APIs, but
not exactly the same. This method is compatible with both approaches but some parameters are only available for
text-generation-inference. If some parameters are ignored, a warning message is triggered but the process
continues correctly.
To learn more about the TGI project, please refer to https://github.com/huggingface/text-generation-inference.
If you want to generate a response from chat messages, you should use the InferenceClient.chat_completion() method. It accepts a list of messages instead of a single text prompt and handles the chat templating for you.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
# Case 1: generate text
>>> await client.text_generation("The huggingface_hub library is ", max_new_tokens=12)
'100% open source and built to be easy to use.'
# Case 2: iterate over the generated tokens. Useful for large generation.
>>> async for token in await client.text_generation("The huggingface_hub library is ", max_new_tokens=12, stream=True):
... print(token)
100
%
open
source
and
built
to
be
easy
to
use
.
# Case 3: get more details about the generation process.
>>> await client.text_generation("The huggingface_hub library is ", max_new_tokens=12, details=True)
TextGenerationOutput(
generated_text='100% open source and built to be easy to use.',
details=TextGenerationDetails(
finish_reason='length',
generated_tokens=12,
seed=None,
prefill=[
TextGenerationPrefillOutputToken(id=487, text='The', logprob=None),
TextGenerationPrefillOutputToken(id=53789, text=' hugging', logprob=-13.171875),
(...)
TextGenerationPrefillOutputToken(id=204, text=' ', logprob=-7.0390625)
],
tokens=[
TokenElement(id=1425, text='100', logprob=-1.0175781, special=False),
TokenElement(id=16, text='%', logprob=-0.0463562, special=False),
(...)
TokenElement(id=25, text='.', logprob=-0.5703125, special=False)
],
best_of_sequences=None
)
)
# Case 4: iterate over the generated tokens with more details.
# Last object is more complete, containing the full generated text and the finish reason.
>>> async for details in await client.text_generation("The huggingface_hub library is ", max_new_tokens=12, details=True, stream=True):
... print(details)
...
TextGenerationStreamOutput(token=TokenElement(id=1425, text='100', logprob=-1.0175781, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=16, text='%', logprob=-0.0463562, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=1314, text=' open', logprob=-1.3359375, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=3178, text=' source', logprob=-0.28100586, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=273, text=' and', logprob=-0.5961914, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=3426, text=' built', logprob=-1.9423828, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=271, text=' to', logprob=-1.4121094, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=314, text=' be', logprob=-1.5224609, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=1833, text=' easy', logprob=-2.1132812, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=271, text=' to', logprob=-0.08520508, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(id=745, text=' use', logprob=-0.39453125, special=False), generated_text=None, details=None)
TextGenerationStreamOutput(token=TokenElement(
id=25,
text='.',
logprob=-0.5703125,
special=False),
generated_text='100% open source and built to be easy to use.',
details=TextGenerationStreamOutputStreamDetails(finish_reason='length', generated_tokens=12, seed=None)
)
# Case 5: generate constrained output using grammar
>>> response = await client.text_generation(
... prompt="I saw a puppy a cat and a raccoon during my bike ride in the park",
... model="HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1",
... max_new_tokens=100,
... repetition_penalty=1.3,
... grammar={
... "type": "json",
... "value": {
... "properties": {
... "location": {"type": "string"},
... "activity": {"type": "string"},
... "animals_seen": {"type": "integer", "minimum": 1, "maximum": 5},
... "animals": {"type": "array", "items": {"type": "string"}},
... },
... "required": ["location", "activity", "animals_seen", "animals"],
... },
... },
... )
>>> json.loads(response)
{
"activity": "bike riding",
"animals": ["puppy", "cat", "raccoon"],
"animals_seen": 3,
"location": "park"
}text_to_image
< source >( prompt: str negative_prompt: typing.Optional[typing.List[str]] = None height: typing.Optional[float] = None width: typing.Optional[float] = None num_inference_steps: typing.Optional[int] = None guidance_scale: typing.Optional[float] = None model: typing.Optional[str] = None scheduler: typing.Optional[str] = None target_size: typing.Optional[huggingface_hub.inference._generated.types.text_to_image.TextToImageTargetSize] = None seed: typing.Optional[int] = None **kwargs ) → Image
Parameters
- prompt (
str) — The prompt to generate an image from. - negative_prompt (
List[str, optional) — One or several prompt to guide what NOT to include in image generation. - height (
float, optional) — The height in pixels of the image to generate. - width (
float, optional) — The width in pixels of the image to generate. - num_inference_steps (
int, optional) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. - guidance_scale (
float, optional) — A higher guidance scale value encourages the model to generate images closely linked to the text prompt, but values too high may cause saturation and other artifacts. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended text-to-image model will be used. Defaults to None. - scheduler (
str, optional) — Override the scheduler with a compatible one. - target_size (
TextToImageTargetSize, optional) — The size in pixel of the output image - seed (
int, optional) — Seed for the random number generator.
Returns
Image
The generated image.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Generate an image based on a given text using a specified model.
You must have PIL installed if you want to work with images (pip install Pillow).
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> image = await client.text_to_image("An astronaut riding a horse on the moon.")
>>> image.save("astronaut.png")
>>> image = await client.text_to_image(
... "An astronaut riding a horse on the moon.",
... negative_prompt="low resolution, blurry",
... model="stabilityai/stable-diffusion-2-1",
... )
>>> image.save("better_astronaut.png")text_to_speech
< source >( text: str model: typing.Optional[str] = None do_sample: typing.Optional[bool] = None early_stopping: typing.Union[bool, ForwardRef('TextToSpeechEarlyStoppingEnum'), NoneType] = None epsilon_cutoff: typing.Optional[float] = None eta_cutoff: typing.Optional[float] = None max_length: typing.Optional[int] = None max_new_tokens: typing.Optional[int] = None min_length: typing.Optional[int] = None min_new_tokens: typing.Optional[int] = None num_beam_groups: typing.Optional[int] = None num_beams: typing.Optional[int] = None penalty_alpha: typing.Optional[float] = None temperature: typing.Optional[float] = None top_k: typing.Optional[int] = None top_p: typing.Optional[float] = None typical_p: typing.Optional[float] = None use_cache: typing.Optional[bool] = None ) → bytes
Parameters
- text (
str) — The text to synthesize. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended text-to-speech model will be used. Defaults to None. - do_sample (
bool, optional) — Whether to use sampling instead of greedy decoding when generating new tokens. - early_stopping (
Union[bool, "TextToSpeechEarlyStoppingEnum", optional) — Controls the stopping condition for beam-based methods. - epsilon_cutoff (
float, optional) — If set to float strictly between 0 and 1, only tokens with a conditional probability greater than epsilon_cutoff will be sampled. In the paper, suggested values range from 3e-4 to 9e-4, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details. - eta_cutoff (
float, optional) — Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than either eta_cutoff or sqrt(eta_cutoff)- exp(-entropy(softmax(next_token_logits))). The latter term is intuitively the expected next token probability, scaled by sqrt(eta_cutoff). In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details. float strictly between 0 and 1, a token is only considered if it is greater than either
- eta_cutoff (
float, optional) — Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than either eta_cutoff or sqrt(eta_cutoff)- exp(-entropy(softmax(next_token_logits))). The latter term is intuitively the expected next token probability, scaled by sqrt(eta_cutoff). In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details.
- max_length (
int, optional) — The maximum length (in tokens) of the generated text, including the input. - max_new_tokens (
int, optional) — The maximum number of tokens to generate. Takes precedence over maxLength. - min_length (
int, optional) — The minimum length (in tokens) of the generated text, including the input. - min_new_tokens (
int, optional) — The minimum number of tokens to generate. Takes precedence over maxLength. - num_beam_groups (
int, optional) — Number of groups to divide num_beams into in order to ensure diversity among different groups of beams. See this paper for more details. - num_beams (
int, optional) — Number of beams to use for beam search. - penalty_alpha (
float, optional) — The value balances the model confidence and the degeneration penalty in contrastive search decoding. - temperature (
float, optional) — The value used to modulate the next token probabilities. - top_k (
int, optional) — The number of highest probability vocabulary tokens to keep for top-k-filtering. - top_p (
float, optional) — If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. - typical_p (
float, optional) — Local typicality measures how similar the conditional probability of predicting a target token next is to the expected conditional probability of predicting a random token next, given the partial text already generated. If set to float < 1, the smallest set of the most locally typical tokens with probabilities that add up to typical_p or higher are kept for generation. See this paper for more details. - use_cache (
bool, optional) — Whether the model should use the past last key/values attentions to speed up decoding
Returns
bytes
The generated audio.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Synthesize an audio of a voice pronouncing a given text.
token_classification
< source >( text: str model: typing.Optional[str] = None aggregation_strategy: typing.Optional[ForwardRef('TokenClassificationAggregationStrategy')] = None ignore_labels: typing.Optional[typing.List[str]] = None stride: typing.Optional[int] = None ) → List[TokenClassificationOutputElement]
Parameters
- text (
str) — A string to be classified. - model (
str, optional) — The model to use for the token classification task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended token classification model will be used. Defaults to None. - aggregation_strategy (
"TokenClassificationAggregationStrategy", optional) — The strategy used to fuse tokens based on model predictions - ignore_labels (
List[str, optional) — A list of labels to ignore - stride (
int, optional) — The number of overlapping tokens between chunks when splitting the input text.
Returns
List[TokenClassificationOutputElement]
List of TokenClassificationOutputElement items containing the entity group, confidence score, word, start and end index.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Perform token classification on the given text. Usually used for sentence parsing, either grammatical, or Named Entity Recognition (NER) to understand keywords contained within text.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.token_classification("My name is Sarah Jessica Parker but you can call me Jessica")
[
TokenClassificationOutputElement(
entity_group='PER',
score=0.9971321225166321,
word='Sarah Jessica Parker',
start=11,
end=31,
),
TokenClassificationOutputElement(
entity_group='PER',
score=0.9773476123809814,
word='Jessica',
start=52,
end=59,
)
]translation
< source >( text: str model: typing.Optional[str] = None src_lang: typing.Optional[str] = None tgt_lang: typing.Optional[str] = None clean_up_tokenization_spaces: typing.Optional[bool] = None truncation: typing.Optional[ForwardRef('TranslationTruncationStrategy')] = None generate_parameters: typing.Optional[typing.Dict[str, typing.Any]] = None ) → TranslationOutput
Parameters
- text (
str) — A string to be translated. - model (
str, optional) — The model to use for the translation task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended translation model will be used. Defaults to None. - src_lang (
str, optional) — The source language of the text. Required for models that can translate from multiple languages. - tgt_lang (
str, optional) — Target language to translate to. Required for models that can translate to multiple languages. - clean_up_tokenization_spaces (
bool, optional) — Whether to clean up the potential extra spaces in the text output. - truncation (
"TranslationTruncationStrategy", optional) — The truncation strategy to use. - generate_parameters (
Dict[str, Any], optional) — Additional parametrization of the text generation algorithm.
Returns
The generated translated text.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError or ValueError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.ValueError— If only one of thesrc_langandtgt_langarguments are provided.
Convert text from one language to another.
Check out https://huggingface.co/tasks/translation for more information on how to choose the best model for
your specific use case. Source and target languages usually depend on the model.
However, it is possible to specify source and target languages for certain models. If you are working with one of these models,
you can use src_lang and tgt_lang arguments to pass the relevant information.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.translation("My name is Wolfgang and I live in Berlin")
'Mein Name ist Wolfgang und ich lebe in Berlin.'
>>> await client.translation("My name is Wolfgang and I live in Berlin", model="Helsinki-NLP/opus-mt-en-fr")
TranslationOutput(translation_text='Je m'appelle Wolfgang et je vis à Berlin.')visual_question_answering
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] question: str model: typing.Optional[str] = None top_k: typing.Optional[int] = None ) → List[VisualQuestionAnsweringOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image for the context. It can be raw bytes, an image file, or a URL to an online image. - question (
str) — Question to be answered. - model (
str, optional) — The model to use for the visual question answering task. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. If not provided, the default recommended visual question answering model will be used. Defaults to None. - top_k (
int, optional) — The number of answers to return (will be chosen by order of likelihood). Note that we return less than topk answers if there are not enough options available within the context.
Returns
List[VisualQuestionAnsweringOutputElement]
a list of VisualQuestionAnsweringOutputElement items containing the predicted label and associated probability.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
InferenceTimeoutError— If the model is unavailable or the request times out.aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Answering open-ended questions based on an image.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.visual_question_answering(
... image="https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg",
... question="What is the animal doing?"
... )
[
VisualQuestionAnsweringOutputElement(score=0.778609573841095, answer='laying down'),
VisualQuestionAnsweringOutputElement(score=0.6957435607910156, answer='sitting'),
]zero_shot_classification
< source >( text: str candidate_labels: typing.List[str] = None multi_label: typing.Optional[bool] = False hypothesis_template: typing.Optional[str] = None model: typing.Optional[str] = None labels: typing.List[str] = None ) → List[ZeroShotClassificationOutputElement]
Parameters
- text (
str) — The input text to classify. - candidate_labels (
List[str]) — The set of possible class labels to classify the text into. - labels (
List[str], optional) — (deprecated) List of strings. Each string is the verbalization of a possible label for the input text. - multi_label (
bool, optional) — Whether multiple candidate labels can be true. If false, the scores are normalized such that the sum of the label likelihoods for each sequence is 1. If true, the labels are considered independent and probabilities are normalized for each candidate. - hypothesis_template (
str, optional) — The sentence used in conjunction with candidateLabels to attempt the text classification by replacing the placeholder with the candidate labels. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. If not provided, the default recommended zero-shot classification model will be used.
Returns
List[ZeroShotClassificationOutputElement]
List of ZeroShotClassificationOutputElement items containing the predicted labels and their confidence.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Provide as input a text and a set of candidate labels to classify the input text.
Example with multi_label=False:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> text = (
... "A new model offers an explanation for how the Galilean satellites formed around the solar system's"
... "largest world. Konstantin Batygin did not set out to solve one of the solar system's most puzzling"
... " mysteries when he went for a run up a hill in Nice, France."
... )
>>> labels = ["space & cosmos", "scientific discovery", "microbiology", "robots", "archeology"]
>>> await client.zero_shot_classification(text, labels)
[
ZeroShotClassificationOutputElement(label='scientific discovery', score=0.7961668968200684),
ZeroShotClassificationOutputElement(label='space & cosmos', score=0.18570658564567566),
ZeroShotClassificationOutputElement(label='microbiology', score=0.00730885099619627),
ZeroShotClassificationOutputElement(label='archeology', score=0.006258360575884581),
ZeroShotClassificationOutputElement(label='robots', score=0.004559356719255447),
]
>>> await client.zero_shot_classification(text, labels, multi_label=True)
[
ZeroShotClassificationOutputElement(label='scientific discovery', score=0.9829297661781311),
ZeroShotClassificationOutputElement(label='space & cosmos', score=0.755190908908844),
ZeroShotClassificationOutputElement(label='microbiology', score=0.0005462635890580714),
ZeroShotClassificationOutputElement(label='archeology', score=0.00047131875180639327),
ZeroShotClassificationOutputElement(label='robots', score=0.00030448526376858354),
]Example with multi_label=True and a custom hypothesis_template:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.zero_shot_classification(
... text="I really like our dinner and I'm very happy. I don't like the weather though.",
... labels=["positive", "negative", "pessimistic", "optimistic"],
... multi_label=True,
... hypothesis_template="This text is {} towards the weather"
... )
[
ZeroShotClassificationOutputElement(label='negative', score=0.9231801629066467),
ZeroShotClassificationOutputElement(label='pessimistic', score=0.8760990500450134),
ZeroShotClassificationOutputElement(label='optimistic', score=0.0008674879791215062),
ZeroShotClassificationOutputElement(label='positive', score=0.0005250611575320363)
]zero_shot_image_classification
< source >( image: typing.Union[bytes, typing.BinaryIO, str, pathlib.Path] candidate_labels: typing.Optional[typing.List[str]] = None model: typing.Optional[str] = None hypothesis_template: typing.Optional[str] = None labels: typing.Optional[typing.List[str]] = None ) → List[ZeroShotImageClassificationOutputElement]
Parameters
- image (
Union[str, Path, bytes, BinaryIO]) — The input image to caption. It can be raw bytes, an image file, or a URL to an online image. - candidate_labels (
List[str]) — The candidate labels for this image - labels (
List[str], optional) — (deprecated) List of string possible labels. There must be at least 2 labels. - model (
str, optional) — The model to use for inference. Can be a model ID hosted on the Hugging Face Hub or a URL to a deployed Inference Endpoint. This parameter overrides the model defined at the instance level. If not provided, the default recommended zero-shot image classification model will be used. - hypothesis_template (
str, optional) — The sentence used in conjunction with candidateLabels to attempt the text classification by replacing the placeholder with the candidate labels.
Returns
List[ZeroShotImageClassificationOutputElement]
List of ZeroShotImageClassificationOutputElement items containing the predicted labels and their confidence.
Raises
InferenceTimeoutError or aiohttp.ClientResponseError
- InferenceTimeoutError — If the model is unavailable or the request times out.
aiohttp.ClientResponseError— If the request fails with an HTTP error status code other than HTTP 503.
Provide input image and text labels to predict text labels for the image.
Example:
# Must be run in an async context
>>> from huggingface_hub import AsyncInferenceClient
>>> client = AsyncInferenceClient()
>>> await client.zero_shot_image_classification(
... "https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Cute_dog.jpg/320px-Cute_dog.jpg",
... labels=["dog", "cat", "horse"],
... )
[ZeroShotImageClassificationOutputElement(label='dog', score=0.956),...]InferenceTimeoutError
Error raised when a model is unavailable or the request times out.
ModelStatus
class huggingface_hub.inference._common.ModelStatus
< source >( loaded: bool state: str compute_type: typing.Dict framework: str )
Parameters
- loaded (
bool) — If the model is currently loaded into Hugging Face’s InferenceAPI. Models are loaded on-demand, leading to the user’s first request taking longer. If a model is loaded, you can be assured that it is in a healthy state. - state (
str) — The current state of the model. This can be ‘Loaded’, ‘Loadable’, ‘TooBig’. If a model’s state is ‘Loadable’, it’s not too big and has a supported backend. Loadable models are automatically loaded when the user first requests inference on the endpoint. This means it is transparent for the user to load a model, except that the first call takes longer to complete. - compute_type (
Dict) — Information about the compute resource the model is using or will use, such as ‘gpu’ type and number of replicas. - framework (
str) — The name of the framework that the model was built with, such as ‘transformers’ or ‘text-generation-inference’.
This Dataclass represents the model status in the Hugging Face Inference API.
InferenceAPI
InferenceAPI is the legacy way to call the Inference API. The interface is more simplistic and requires knowing
the input parameters and output format for each task. It also lacks the ability to connect to other services like
Inference Endpoints or AWS SageMaker. InferenceAPI will soon be deprecated so we recommend using InferenceClient
whenever possible. Check out this guide to learn how to switch from
InferenceAPI to InferenceClient in your scripts.
class huggingface_hub.InferenceApi
< source >( repo_id: str task: typing.Optional[str] = None token: typing.Optional[str] = None gpu: bool = False )
Client to configure requests and make calls to the HuggingFace Inference API.
Example:
>>> from huggingface_hub.inference_api import InferenceApi
>>> # Mask-fill example
>>> inference = InferenceApi("bert-base-uncased")
>>> inference(inputs="The goal of life is [MASK].")
[{'sequence': 'the goal of life is life.', 'score': 0.10933292657136917, 'token': 2166, 'token_str': 'life'}]
>>> # Question Answering example
>>> inference = InferenceApi("deepset/roberta-base-squad2")
>>> inputs = {
... "question": "What's my name?",
... "context": "My name is Clara and I live in Berkeley.",
... }
>>> inference(inputs)
{'score': 0.9326569437980652, 'start': 11, 'end': 16, 'answer': 'Clara'}
>>> # Zero-shot example
>>> inference = InferenceApi("typeform/distilbert-base-uncased-mnli")
>>> inputs = "Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!"
>>> params = {"candidate_labels": ["refund", "legal", "faq"]}
>>> inference(inputs, params)
{'sequence': 'Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!', 'labels': ['refund', 'faq', 'legal'], 'scores': [0.9378499388694763, 0.04914155602455139, 0.013008488342165947]}
>>> # Overriding configured task
>>> inference = InferenceApi("bert-base-uncased", task="feature-extraction")
>>> # Text-to-image
>>> inference = InferenceApi("stabilityai/stable-diffusion-2-1")
>>> inference("cat")
<PIL.PngImagePlugin.PngImageFile image (...)>
>>> # Return as raw response to parse the output yourself
>>> inference = InferenceApi("mio/amadeus")
>>> response = inference("hello world", raw_response=True)
>>> response.headers
{"Content-Type": "audio/flac", ...}
>>> response.content # raw bytes from server
b'(...)'__init__
< source >( repo_id: str task: typing.Optional[str] = None token: typing.Optional[str] = None gpu: bool = False )
Parameters
- repo_id (
str) — Id of repository (e.g. user/bert-base-uncased). - task (
str, optional, defaultsNone) — Whether to force a task instead of using task specified in the repository. - token (str, optional) — The API token to use as HTTP bearer authorization. This is not the authentication token. You can find the token in https://huggingface.co/settings/token. Alternatively, you can find both your organizations and personal API tokens using HfApi().whoami(token).
- gpu (bool, optional, defaults False) — Whether to use GPU instead of CPU for inference(requires Startup plan at least).
Inits headers and API call information.
__call__
< source >( inputs: typing.Union[str, typing.Dict, typing.List[str], typing.List[typing.List[str]], NoneType] = None params: typing.Optional[typing.Dict] = None data: typing.Optional[bytes] = None raw_response: bool = False )
Parameters
- inputs (
strorDictorList[str]orList[List[str]], optional) — Inputs for the prediction. - params (
Dict, optional) — Additional parameters for the models. Will be sent asparametersin the payload. - data (
bytes, optional) — Bytes content of the request. In this case, leaveinputsandparamsempty. - raw_response (
bool, defaults toFalse) — IfTrue, the rawResponseobject is returned. You can parse its content as preferred. By default, the content is parsed into a more practical format (json dictionary or PIL Image for example).
Make a call to the Inference API.