{value}
\n\n```" }, { "id": 56, "parent": 53, "path": "08_custom-components/05_frontend.md", "level": 2, "title": "The Example.svelte file", "content": "The `Example.svelte` file should expose the following props:\n\n```typescript\n export let value: string;\n export let type: \"gallery\" | \"table\";\n export let selected = false;\n export let index: number;\n```\n\n* `value`: The example value that should be displayed.\n\n* `type`: This is a variable that can be either `\"gallery\"` or `\"table\"` depending on how the examples are displayed. The `\"gallery\"` form is used when the examples correspond to a single input component, while the `\"table\"` form is used when a user has multiple input components, and the examples need to populate all of them. \n\n* `selected`: You can also adjust how the examples are displayed if a user \"selects\" a particular example by using the selected variable.\n\n* `index`: The current index of the selected value.\n\n* Any additional props your \"non-example\" component takes!\n\nThis is the `Example.svelte` file for the code `Radio` component:\n\n```svelte\n\n\n \n
"
},
{
"id": 122,
"parent": 121,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "What do I need to add hints to?",
"content": "You do not need to add type hints to every part of your code. For the documentation to work correctly, you will need to add type hints to the following component methods:\n\n- `__init__` parameters should be typed.\n- `postprocess` parameters and return value should be typed.\n- `preprocess` parameters and return value should be typed.\n\nIf you are using `gradio cc create`, these types should already exist, but you may need to tweak them based on any changes you make."
},
{
"id": 123,
"parent": 122,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 5,
"title": "`__init__`",
"content": "Here, you only need to type the parameters. If you have cloned a template with `gradio` cc create`, these should already be in place. You will only need to add new hints for anything you have added or changed:\n\n```py\ndef __init__(\n self,\n value: str | None = None,\n *,\n sources: Literal[\"upload\", \"microphone\"] = \"upload,\n every: Timer | float | None = None,\n ...\n):\n ...\n```"
},
{
"id": 124,
"parent": 122,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 5,
"title": "`preprocess` and `postprocess`",
"content": "The `preprocess` and `postprocess` methods determine the value passed to the user function and the value that needs to be returned.\n\nEven if the design of your component is primarily as an input or an output, it is worth adding type hints to both the input parameters and the return values because Gradio has no way of limiting how components can be used.\n\nIn this case, we specifically care about:\n\n- The return type of `preprocess`.\n- The input type of `postprocess`.\n\n```py\ndef preprocess(\n self, payload: FileData | None # input is optional\n) -> tuple[int, str] | str | None:"
},
{
"id": 125,
"parent": null,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 1,
"title": "user function input is the preprocess return ▲",
"content": ""
},
{
"id": 126,
"parent": null,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 1,
"title": "user function output is the postprocess input ▼",
"content": "def postprocess(\n self, value: tuple[int, str] | None\n) -> FileData | bytes | None: # return is optional\n ...\n```"
},
{
"id": 127,
"parent": 126,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 3,
"title": "Docstrings",
"content": "Docstrings are also used extensively to extract more meaningful, human-readable descriptions of certain parts of the API.\n\nWhat are type hints?
\n\n\nIf you need to become more familiar with type hints in Python, they are a simple way to express what Python types are expected for arguments and return values of functions and methods. They provide a helpful in-editor experience, aid in maintenance, and integrate with various other tools. These types can be simple primitives, like `list` `str` `bool`; they could be more compound types like `list[str]`, `str | None` or `tuple[str, float | int]`; or they can be more complex types using utility classed like [`TypedDict`](https://peps.python.org/pep-0589/#abstract).\n\n[Read more about type hints in Python.](https://realpython.com/lessons/type-hinting/)\n\n\n \n
\n\nWhile docstrings don't have any syntax requirements, we need a particular structure for documentation purposes.\n\nAs with type hint, the specific information we care about is as follows:\n\n- `__init__` parameter docstrings.\n- `preprocess` return docstrings.\n- `postprocess` input parameter docstrings.\n\nEverything else is optional.\n\nDocstrings should always take this format to be picked up by the documentation generator:"
},
{
"id": 128,
"parent": 127,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Classes",
"content": "```py\n\"\"\"\nA description of the class.\n\nThis can span multiple lines and can _contain_ *markdown*.\n\"\"\"\n```"
},
{
"id": 129,
"parent": 127,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Methods and functions ",
"content": "Markdown in these descriptions will not be converted into formatted text.\n\n```py\n\"\"\"\nParameters:\n param_one: A description for this parameter.\n param_two: A description for this parameter.\nReturns:\n A description for this return value.\n\"\"\"\n```"
},
{
"id": 130,
"parent": 126,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 3,
"title": "Events",
"content": "In custom components, events are expressed as a list stored on the `events` field of the component class. While we do not need types for events, we _do_ need a human-readable description so users can understand the behaviour of the event.\n\nTo facilitate this, we must create the event in a specific way.\n\nThere are two ways to add events to a custom component."
},
{
"id": 131,
"parent": 130,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Built-in events",
"content": "Gradio comes with a variety of built-in events that may be enough for your component. If you are using built-in events, you do not need to do anything as they already have descriptions we can extract:\n\n```py\nfrom gradio.events import Events\n\nclass ParamViewer(Component):\n ...\n\n EVENTS = [\n Events.change,\n Events.upload,\n ]\n```"
},
{
"id": 132,
"parent": 130,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Custom events",
"content": "You can define a custom event if the built-in events are unsuitable for your use case. This is a straightforward process, but you must create the event in this way for docstrings to work correctly:\n\n```py\nfrom gradio.events import Events, EventListener\n\nclass ParamViewer(Component):\n ...\n\n EVENTS = [\n Events.change,\n EventListener(\n \"bingbong\",\n doc=\"This listener is triggered when the user does a bingbong.\"\n )\n ]\n```"
},
{
"id": 133,
"parent": 126,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 3,
"title": "Demo",
"content": "The `demo/app.py`, often used for developing the component, generates the live demo and code snippet. The only strict rule here is that the `demo.launch()` command must be contained with a `__name__ == \"__main__\"` conditional as below:\n\n```py\nif __name__ == \"__main__\":\n demo.launch()\n```\n\nThe documentation generator will scan for such a clause and error if absent. If you are _not_ launching the demo inside the `demo/app.py`, then you can pass `--suppress-demo-check` to turn off this check."
},
{
"id": 134,
"parent": 133,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Demo recommendations",
"content": "Although there are no additional rules, there are some best practices you should bear in mind to get the best experience from the documentation generator.\n\nThese are only guidelines, and every situation is unique, but they are sound principles to remember."
},
{
"id": 135,
"parent": 134,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 5,
"title": "Keep the demo compact",
"content": "Compact demos look better and make it easier for users to understand what the demo does. Try to remove as many extraneous UI elements as possible to focus the users' attention on the core use case. \n\nSometimes, it might make sense to have a `demo/app.py` just for the docs and an additional, more complex app for your testing purposes. You can also create other spaces, showcasing more complex examples and linking to them from the main class docstring or the `pyproject.toml` description."
},
{
"id": 136,
"parent": 133,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Keep the code concise",
"content": "The 'getting started' snippet utilises the demo code, which should be as short as possible to keep users engaged and avoid confusion.\n\nIt isn't the job of the sample snippet to demonstrate the whole API; this snippet should be the shortest path to success for a new user. It should be easy to type or copy-paste and easy to understand. Explanatory comments should be brief and to the point."
},
{
"id": 137,
"parent": 133,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Avoid external dependencies",
"content": "As mentioned above, users should be able to copy-paste a snippet and have a fully working app. Try to avoid third-party library dependencies to facilitate this.\n\nYou should carefully consider any examples; avoiding examples that require additional files or that make assumptions about the environment is generally a good idea."
},
{
"id": 138,
"parent": 133,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 4,
"title": "Ensure the `demo` directory is self-contained",
"content": "Only the `demo` directory will be uploaded to Hugging Face spaces in certain instances, as the component will be installed via PyPi if possible. It is essential that this directory is self-contained and any files needed for the correct running of the demo are present."
},
{
"id": 139,
"parent": 126,
"path": "08_custom-components/09_documenting-custom-components.md",
"level": 3,
"title": "Additional URLs",
"content": "The documentation generator will generate a few buttons, providing helpful information and links to users. They are obtained automatically in some cases, but some need to be explicitly included in the `pyproject.yaml`. \n\n- PyPi Version and link - This is generated automatically.\n- GitHub Repository - This is populated via the `pyproject.toml`'s `project.urls.repository`.\n- Hugging Face Space - This is populated via the `pyproject.toml`'s `project.urls.space`.\n\nAn example `pyproject.toml` urls section might look like this:\n\n```toml\n[project.urls]\nrepository = \"https://github.com/user/repo-name\"\nspace = \"https://huggingface.co/spaces/user/space-name\"\n```"
},
{
"id": 140,
"parent": null,
"path": "08_custom-components/03_configuration.md",
"level": 1,
"title": "Configuring Your Custom Component",
"content": "The custom components workflow focuses on [convention over configuration](https://en.wikipedia.org/wiki/Convention_over_configuration) to reduce the number of decisions you as a developer need to make when developing your custom component.\nThat being said, you can still configure some aspects of the custom component package and directory.\nThis guide will cover how."
},
{
"id": 141,
"parent": 140,
"path": "08_custom-components/03_configuration.md",
"level": 2,
"title": "The Package Name",
"content": "By default, all custom component packages are called `gradio_What are docstrings?
\n\n\nIf you need to become more familiar with docstrings in Python, they are a way to annotate parts of your code with human-readable decisions and explanations. They offer a rich in-editor experience like type hints, but unlike type hints, they don't have any specific syntax requirements. They are simple strings and can take almost any form. The only requirement is where they appear. Docstrings should be \"a string literal that occurs as the first statement in a module, function, class, or method definition\".\n\n[Read more about Python docstrings.](https://peps.python.org/pep-0257/#what-is-a-docstring)\n\n\n\n
\n\nGradio comes with a set of prebuilt themes which you can load from `gr.themes.*`. These are:\n\n\n* `gr.themes.Base()` - the `\"base\"` theme sets the primary color to blue but otherwise has minimal styling, making it particularly useful as a base for creating new, custom themes.\n* `gr.themes.Default()` - the `\"default\"` Gradio 5 theme, with a vibrant orange primary color and gray secondary color.\n* `gr.themes.Origin()` - the `\"origin\"` theme is most similar to Gradio 4 styling. Colors, especially in light mode, are more subdued than the Gradio 5 default theme.\n* `gr.themes.Citrus()` - the `\"citrus\"` theme uses a yellow primary color, highlights form elements that are in focus, and includes fun 3D effects when buttons are clicked.\n* `gr.themes.Monochrome()` - the `\"monochrome\"` theme uses a black primary and white secondary color, and uses serif-style fonts, giving the appearance of a black-and-white newspaper. \n* `gr.themes.Soft()` - the `\"soft\"` theme uses a purpose primary color and white secondary color. It also increases the border radii and around buttons and form elements and highlights labels.\n* `gr.themes.Glass()` - the `\"glass\"` theme has a blue primary color and a transclucent gray secondary color. The theme also uses vertical gradients to create a glassy effect.\n* `gr.themes.Ocean()` - the `\"ocean\"` theme has a blue-green primary color and gray secondary color. The theme also uses horizontal gradients, especially for buttons and some form elements.\n\n\nEach of these themes set values for hundreds of CSS variables. You can use prebuilt themes as a starting point for your own custom themes, or you can create your own themes from scratch. Let's take a look at each approach."
},
{
"id": 183,
"parent": 181,
"path": "10_other-tutorials/theming-guide.md",
"level": 2,
"title": "Using the Theme Builder",
"content": "The easiest way to build a theme is using the Theme Builder. To launch the Theme Builder locally, run the following code:\n\n```python\nimport gradio as gr\n\ngr.themes.builder()\n```\n\n$demo_theme_builder\n\nYou can use the Theme Builder running on Spaces above, though it runs much faster when you launch it locally via `gr.themes.builder()`.\n\nAs you edit the values in the Theme Builder, the app will preview updates in real time. You can download the code to generate the theme you've created so you can use it in any Gradio app.\n\nIn the rest of the guide, we will cover building themes programmatically."
},
{
"id": 184,
"parent": 181,
"path": "10_other-tutorials/theming-guide.md",
"level": 2,
"title": "Extending Themes via the Constructor",
"content": "Although each theme has hundreds of CSS variables, the values for most these variables are drawn from 8 core variables which can be set through the constructor of each prebuilt theme. Modifying these 8 arguments allows you to quickly change the look and feel of your app."
},
{
"id": 185,
"parent": 184,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "Core Colors",
"content": "The first 3 constructor arguments set the colors of the theme and are `gradio.themes.Color` objects. Internally, these Color objects hold brightness values for the palette of a single hue, ranging from 50, 100, 200..., 800, 900, 950. Other CSS variables are derived from these 3 colors.\n\nThe 3 color constructor arguments are:\n\n- `primary_hue`: This is the color draws attention in your theme. In the default theme, this is set to `gradio.themes.colors.orange`.\n- `secondary_hue`: This is the color that is used for secondary elements in your theme. In the default theme, this is set to `gradio.themes.colors.blue`.\n- `neutral_hue`: This is the color that is used for text and other neutral elements in your theme. In the default theme, this is set to `gradio.themes.colors.gray`.\n\nYou could modify these values using their string shortcuts, such as\n\n```python\nwith gr.Blocks(theme=gr.themes.Default(primary_hue=\"red\", secondary_hue=\"pink\")) as demo:\n ...\n```\n\nor you could use the `Color` objects directly, like this:\n\n```python\nwith gr.Blocks(theme=gr.themes.Default(primary_hue=gr.themes.colors.red, secondary_hue=gr.themes.colors.pink)) as demo:\n ...\n```\n\n\n\n
\n\nPredefined colors are:\n\n- `slate`\n- `gray`\n- `zinc`\n- `neutral`\n- `stone`\n- `red`\n- `orange`\n- `amber`\n- `yellow`\n- `lime`\n- `green`\n- `emerald`\n- `teal`\n- `cyan`\n- `sky`\n- `blue`\n- `indigo`\n- `violet`\n- `purple`\n- `fuchsia`\n- `pink`\n- `rose`\n\nYou could also create your own custom `Color` objects and pass them in."
},
{
"id": 186,
"parent": 184,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "Core Sizing",
"content": "The next 3 constructor arguments set the sizing of the theme and are `gradio.themes.Size` objects. Internally, these Size objects hold pixel size values that range from `xxs` to `xxl`. Other CSS variables are derived from these 3 sizes.\n\n- `spacing_size`: This sets the padding within and spacing between elements. In the default theme, this is set to `gradio.themes.sizes.spacing_md`.\n- `radius_size`: This sets the roundedness of corners of elements. In the default theme, this is set to `gradio.themes.sizes.radius_md`.\n- `text_size`: This sets the font size of text. In the default theme, this is set to `gradio.themes.sizes.text_md`.\n\nYou could modify these values using their string shortcuts, such as\n\n```python\nwith gr.Blocks(theme=gr.themes.Default(spacing_size=\"sm\", radius_size=\"none\")) as demo:\n ...\n```\n\nor you could use the `Size` objects directly, like this:\n\n```python\nwith gr.Blocks(theme=gr.themes.Default(spacing_size=gr.themes.sizes.spacing_sm, radius_size=gr.themes.sizes.radius_none)) as demo:\n ...\n```\n\n\n\n
\n\nThe predefined size objects are:\n\n- `radius_none`\n- `radius_sm`\n- `radius_md`\n- `radius_lg`\n- `spacing_sm`\n- `spacing_md`\n- `spacing_lg`\n- `text_sm`\n- `text_md`\n- `text_lg`\n\nYou could also create your own custom `Size` objects and pass them in."
},
{
"id": 187,
"parent": 184,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "Core Fonts",
"content": "The final 2 constructor arguments set the fonts of the theme. You can pass a list of fonts to each of these arguments to specify fallbacks. If you provide a string, it will be loaded as a system font. If you provide a `gradio.themes.GoogleFont`, the font will be loaded from Google Fonts.\n\n- `font`: This sets the primary font of the theme. In the default theme, this is set to `gradio.themes.GoogleFont(\"IBM Plex Sans\")`.\n- `font_mono`: This sets the monospace font of the theme. In the default theme, this is set to `gradio.themes.GoogleFont(\"IBM Plex Mono\")`.\n\nYou could modify these values such as the following:\n\n```python\nwith gr.Blocks(theme=gr.themes.Default(font=[gr.themes.GoogleFont(\"Inconsolata\"), \"Arial\", \"sans-serif\"])) as demo:\n ...\n```\n\n\n\n
"
},
{
"id": 188,
"parent": 181,
"path": "10_other-tutorials/theming-guide.md",
"level": 2,
"title": "Extending Themes via `.set()`",
"content": "You can also modify the values of CSS variables after the theme has been loaded. To do so, use the `.set()` method of the theme object to get access to the CSS variables. For example:\n\n```python\ntheme = gr.themes.Default(primary_hue=\"blue\").set(\n loader_color=\"#FF0000\",\n slider_color=\"#FF0000\",\n)\n\nwith gr.Blocks(theme=theme) as demo:\n ...\n```\n\nIn the example above, we've set the `loader_color` and `slider_color` variables to `#FF0000`, despite the overall `primary_color` using the blue color palette. You can set any CSS variable that is defined in the theme in this manner.\n\nYour IDE type hinting should help you navigate these variables. Since there are so many CSS variables, let's take a look at how these variables are named and organized."
},
{
"id": 189,
"parent": 188,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "CSS Variable Naming Conventions",
"content": "CSS variable names can get quite long, like `button_primary_background_fill_hover_dark`! However they follow a common naming convention that makes it easy to understand what they do and to find the variable you're looking for. Separated by underscores, the variable name is made up of:\n\n1. The target element, such as `button`, `slider`, or `block`.\n2. The target element type or sub-element, such as `button_primary`, or `block_label`.\n3. The property, such as `button_primary_background_fill`, or `block_label_border_width`.\n4. Any relevant state, such as `button_primary_background_fill_hover`.\n5. If the value is different in dark mode, the suffix `_dark`. For example, `input_border_color_focus_dark`.\n\nOf course, many CSS variable names are shorter than this, such as `table_border_color`, or `input_shadow`."
},
{
"id": 190,
"parent": 188,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "CSS Variable Organization",
"content": "Though there are hundreds of CSS variables, they do not all have to have individual values. They draw their values by referencing a set of core variables and referencing each other. This allows us to only have to modify a few variables to change the look and feel of the entire theme, while also getting finer control of individual elements that we may want to modify."
},
{
"id": 191,
"parent": 190,
"path": "10_other-tutorials/theming-guide.md",
"level": 4,
"title": "Referencing Core Variables",
"content": "To reference one of the core constructor variables, precede the variable name with an asterisk. To reference a core color, use the `*primary_`, `*secondary_`, or `*neutral_` prefix, followed by the brightness value. For example:\n\n```python\ntheme = gr.themes.Default(primary_hue=\"blue\").set(\n button_primary_background_fill=\"*primary_200\",\n button_primary_background_fill_hover=\"*primary_300\",\n)\n```\n\nIn the example above, we've set the `button_primary_background_fill` and `button_primary_background_fill_hover` variables to `*primary_200` and `*primary_300`. These variables will be set to the 200 and 300 brightness values of the blue primary color palette, respectively.\n\nSimilarly, to reference a core size, use the `*spacing_`, `*radius_`, or `*text_` prefix, followed by the size value. For example:\n\n```python\ntheme = gr.themes.Default(radius_size=\"md\").set(\n button_primary_border_radius=\"*radius_xl\",\n)\n```\n\nIn the example above, we've set the `button_primary_border_radius` variable to `*radius_xl`. This variable will be set to the `xl` setting of the medium radius size range."
},
{
"id": 192,
"parent": 190,
"path": "10_other-tutorials/theming-guide.md",
"level": 4,
"title": "Referencing Other Variables",
"content": "Variables can also reference each other. For example, look at the example below:\n\n```python\ntheme = gr.themes.Default().set(\n button_primary_background_fill=\"#FF0000\",\n button_primary_background_fill_hover=\"#FF0000\",\n button_primary_border=\"#FF0000\",\n)\n```\n\nHaving to set these values to a common color is a bit tedious. Instead, we can reference the `button_primary_background_fill` variable in the `button_primary_background_fill_hover` and `button_primary_border` variables, using a `*` prefix.\n\n```python\ntheme = gr.themes.Default().set(\n button_primary_background_fill=\"#FF0000\",\n button_primary_background_fill_hover=\"*button_primary_background_fill\",\n button_primary_border=\"*button_primary_background_fill\",\n)\n```\n\nNow, if we change the `button_primary_background_fill` variable, the `button_primary_background_fill_hover` and `button_primary_border` variables will automatically update as well.\n\nThis is particularly useful if you intend to share your theme - it makes it easy to modify the theme without having to change every variable.\n\nNote that dark mode variables automatically reference each other. For example:\n\n```python\ntheme = gr.themes.Default().set(\n button_primary_background_fill=\"#FF0000\",\n button_primary_background_fill_dark=\"#AAAAAA\",\n button_primary_border=\"*button_primary_background_fill\",\n button_primary_border_dark=\"*button_primary_background_fill_dark\",\n)\n```\n\n`button_primary_border_dark` will draw its value from `button_primary_background_fill_dark`, because dark mode always draw from the dark version of the variable."
},
{
"id": 193,
"parent": 181,
"path": "10_other-tutorials/theming-guide.md",
"level": 2,
"title": "Creating a Full Theme",
"content": "Let's say you want to create a theme from scratch! We'll go through it step by step - you can also see the source of prebuilt themes in the gradio source repo for reference - [here's the source](https://github.com/gradio-app/gradio/blob/main/gradio/themes/monochrome.py) for the Monochrome theme.\n\nOur new theme class will inherit from `gradio.themes.Base`, a theme that sets a lot of convenient defaults. Let's make a simple demo that creates a dummy theme called Seafoam, and make a simple app that uses it.\n\n```py\nimport gradio as gr\nfrom gradio.themes.base import Base\nimport time\n\nclass Seafoam(Base):\n pass\n\nseafoam = Seafoam()\n\nwith gr.Blocks(theme=seafoam) as demo:\n textbox = gr.Textbox(label=\"Name\")\n slider = gr.Slider(label=\"Count\", minimum=0, maximum=100, step=1)\n with gr.Row():\n button = gr.Button(\"Submit\", variant=\"primary\")\n clear = gr.Button(\"Clear\")\n output = gr.Textbox(label=\"Output\")\n\n def repeat(name, count):\n time.sleep(3)\n return name * count\n\n button.click(repeat, [textbox, slider], output)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n\n\n\n
\n\nThe Base theme is very barebones, and uses `gr.themes.Blue` as it primary color - you'll note the primary button and the loading animation are both blue as a result. Let's change the defaults core arguments of our app. We'll overwrite the constructor and pass new defaults for the core constructor arguments.\n\nWe'll use `gr.themes.Emerald` as our primary color, and set secondary and neutral hues to `gr.themes.Blue`. We'll make our text larger using `text_lg`. We'll use `Quicksand` as our default font, loaded from Google Fonts.\n\n```py\nfrom __future__ import annotations\nfrom typing import Iterable\nimport gradio as gr\nfrom gradio.themes.base import Base\nfrom gradio.themes.utils import colors, fonts, sizes\nimport time\n\nclass Seafoam(Base):\n def __init__(\n self,\n *,\n primary_hue: colors.Color | str = colors.emerald,\n secondary_hue: colors.Color | str = colors.blue,\n neutral_hue: colors.Color | str = colors.gray,\n spacing_size: sizes.Size | str = sizes.spacing_md,\n radius_size: sizes.Size | str = sizes.radius_md,\n text_size: sizes.Size | str = sizes.text_lg,\n font: fonts.Font\n | str\n | Iterable[fonts.Font | str] = (\n fonts.GoogleFont(\"Quicksand\"),\n \"ui-sans-serif\",\n \"sans-serif\",\n ),\n font_mono: fonts.Font\n | str\n | Iterable[fonts.Font | str] = (\n fonts.GoogleFont(\"IBM Plex Mono\"),\n \"ui-monospace\",\n \"monospace\",\n ),\n ):\n super().__init__(\n primary_hue=primary_hue,\n secondary_hue=secondary_hue,\n neutral_hue=neutral_hue,\n spacing_size=spacing_size,\n radius_size=radius_size,\n text_size=text_size,\n font=font,\n font_mono=font_mono,\n )\n\nseafoam = Seafoam()\n\nwith gr.Blocks(theme=seafoam) as demo:\n textbox = gr.Textbox(label=\"Name\")\n slider = gr.Slider(label=\"Count\", minimum=0, maximum=100, step=1)\n with gr.Row():\n button = gr.Button(\"Submit\", variant=\"primary\")\n clear = gr.Button(\"Clear\")\n output = gr.Textbox(label=\"Output\")\n\n def repeat(name, count):\n time.sleep(3)\n return name * count\n\n button.click(repeat, [textbox, slider], output)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n\n\n\n
\n\nSee how the primary button and the loading animation are now green? These CSS variables are tied to the `primary_hue` variable.\n\nLet's modify the theme a bit more directly. We'll call the `set()` method to overwrite CSS variable values explicitly. We can use any CSS logic, and reference our core constructor arguments using the `*` prefix.\n\n```py\nfrom __future__ import annotations\nfrom typing import Iterable\nimport gradio as gr\nfrom gradio.themes.base import Base\nfrom gradio.themes.utils import colors, fonts, sizes\nimport time\n\nclass Seafoam(Base):\n def __init__(\n self,\n *,\n primary_hue: colors.Color | str = colors.emerald,\n secondary_hue: colors.Color | str = colors.blue,\n neutral_hue: colors.Color | str = colors.blue,\n spacing_size: sizes.Size | str = sizes.spacing_md,\n radius_size: sizes.Size | str = sizes.radius_md,\n text_size: sizes.Size | str = sizes.text_lg,\n font: fonts.Font\n | str\n | Iterable[fonts.Font | str] = (\n fonts.GoogleFont(\"Quicksand\"),\n \"ui-sans-serif\",\n \"sans-serif\",\n ),\n font_mono: fonts.Font\n | str\n | Iterable[fonts.Font | str] = (\n fonts.GoogleFont(\"IBM Plex Mono\"),\n \"ui-monospace\",\n \"monospace\",\n ),\n ):\n super().__init__(\n primary_hue=primary_hue,\n secondary_hue=secondary_hue,\n neutral_hue=neutral_hue,\n spacing_size=spacing_size,\n radius_size=radius_size,\n text_size=text_size,\n font=font,\n font_mono=font_mono,\n )\n super().set(\n body_background_fill=\"repeating-linear-gradient(45deg, *primary_200, *primary_200 10px, *primary_50 10px, *primary_50 20px)\",\n body_background_fill_dark=\"repeating-linear-gradient(45deg, *primary_800, *primary_800 10px, *primary_900 10px, *primary_900 20px)\",\n button_primary_background_fill=\"linear-gradient(90deg, *primary_300, *secondary_400)\",\n button_primary_background_fill_hover=\"linear-gradient(90deg, *primary_200, *secondary_300)\",\n button_primary_text_color=\"white\",\n button_primary_background_fill_dark=\"linear-gradient(90deg, *primary_600, *secondary_800)\",\n slider_color=\"*secondary_300\",\n slider_color_dark=\"*secondary_600\",\n block_title_text_weight=\"600\",\n block_border_width=\"3px\",\n block_shadow=\"*shadow_drop_lg\",\n button_primary_shadow=\"*shadow_drop_lg\",\n button_large_padding=\"32px\",\n )\n\nseafoam = Seafoam()\n\nwith gr.Blocks(theme=seafoam) as demo:\n textbox = gr.Textbox(label=\"Name\")\n slider = gr.Slider(label=\"Count\", minimum=0, maximum=100, step=1)\n with gr.Row():\n button = gr.Button(\"Submit\", variant=\"primary\")\n clear = gr.Button(\"Clear\")\n output = gr.Textbox(label=\"Output\")\n\n def repeat(name, count):\n time.sleep(3)\n return name * count\n\n button.click(repeat, [textbox, slider], output)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n\n\n\n
\n\nLook how fun our theme looks now! With just a few variable changes, our theme looks completely different.\n\nYou may find it helpful to explore the [source code of the other prebuilt themes](https://github.com/gradio-app/gradio/blob/main/gradio/themes) to see how they modified the base theme. You can also find your browser's Inspector useful to select elements from the UI and see what CSS variables are being used in the styles panel."
},
{
"id": 194,
"parent": 181,
"path": "10_other-tutorials/theming-guide.md",
"level": 2,
"title": "Sharing Themes",
"content": "Once you have created a theme, you can upload it to the HuggingFace Hub to let others view it, use it, and build off of it!"
},
{
"id": 195,
"parent": 194,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "Uploading a Theme",
"content": "There are two ways to upload a theme, via the theme class instance or the command line. We will cover both of them with the previously created `seafoam` theme.\n\n- Via the class instance\n\nEach theme instance has a method called `push_to_hub` we can use to upload a theme to the HuggingFace hub.\n\n```python\nseafoam.push_to_hub(repo_name=\"seafoam\",\n version=\"0.0.1\",\n\t\t\t\t\thf_token=\"\n\n
"
},
{
"id": 197,
"parent": 194,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "Discovering Themes",
"content": "The [Theme Gallery](https://huggingface.co/spaces/gradio/theme-gallery) shows all the public gradio themes. After publishing your theme,\nit will automatically show up in the theme gallery after a couple of minutes.\n\nYou can sort the themes by the number of likes on the space and from most to least recently created as well as toggling themes between light and dark mode.\n\n\n\n
"
},
{
"id": 198,
"parent": 194,
"path": "10_other-tutorials/theming-guide.md",
"level": 3,
"title": "Downloading",
"content": "To use a theme from the hub, use the `from_hub` method on the `ThemeClass` and pass it to your app:\n\n```python\nmy_theme = gr.Theme.from_hub(\"gradio/seafoam\")\n\nwith gr.Blocks(theme=my_theme) as demo:\n ....\n```\n\nYou can also pass the theme string directly to `Blocks` or `Interface` (`gr.Blocks(theme=\"gradio/seafoam\")`)\n\nYou can pin your app to an upstream theme version by using semantic versioning expressions.\n\nFor example, the following would ensure the theme we load from the `seafoam` repo was between versions `0.0.1` and `0.1.0`:\n\n```python\nwith gr.Blocks(theme=\"gradio/seafoam@>=0.0.1,<0.1.0\") as demo:\n ....\n```\n\nEnjoy creating your own themes! If you make one you're proud of, please share it with the world by uploading it to the hub!\nIf you tag us on [Twitter](https://twitter.com/gradio) we can give your theme a shout out!\n\n"
},
{
"id": 199,
"parent": null,
"path": "10_other-tutorials/01_using-hugging-face-integrations.md",
"level": 1,
"title": "Using Hugging Face Integrations",
"content": "Related spaces: https://huggingface.co/spaces/gradio/en2es\nTags: HUB, SPACES, EMBED\n\nContributed by Omar Sanseviero 🦙"
},
{
"id": 200,
"parent": 199,
"path": "10_other-tutorials/01_using-hugging-face-integrations.md",
"level": 2,
"title": "Introduction",
"content": "The Hugging Face Hub is a central platform that has hundreds of thousands of [models](https://huggingface.co/models), [datasets](https://huggingface.co/datasets) and [demos](https://huggingface.co/spaces) (also known as Spaces). \n\nGradio has multiple features that make it extremely easy to leverage existing models and Spaces on the Hub. This guide walks through these features."
},
{
"id": 201,
"parent": 199,
"path": "10_other-tutorials/01_using-hugging-face-integrations.md",
"level": 2,
"title": "Demos with the Hugging Face Inference Endpoints",
"content": "Hugging Face has a service called [Serverless Inference Endpoints](https://huggingface.co/docs/api-inference/index), which allows you to send HTTP requests to models on the Hub. The API includes a generous free tier, and you can switch to [dedicated Inference Endpoints](https://huggingface.co/inference-endpoints/dedicated) when you want to use it in production. Gradio integrates directly with Serverless Inference Endpoints so that you can create a demo simply by specifying a model's name (e.g. `Helsinki-NLP/opus-mt-en-es`), like this:\n\n```python\nimport gradio as gr\n\ndemo = gr.load(\"Helsinki-NLP/opus-mt-en-es\", src=\"models\")\n\ndemo.launch()\n```\n\nFor any Hugging Face model supported in Inference Endpoints, Gradio automatically infers the expected input and output and make the underlying server calls, so you don't have to worry about defining the prediction function. \n\nNotice that we just put specify the model name and state that the `src` should be `models` (Hugging Face's Model Hub). There is no need to install any dependencies (except `gradio`) since you are not loading the model on your computer.\n\nYou might notice that the first inference takes a little bit longer. This happens since the Inference Endpoints is loading the model in the server. You get some benefits afterward:\n\n- The inference will be much faster.\n- The server caches your requests.\n- You get built-in automatic scaling."
},
{
"id": 202,
"parent": 199,
"path": "10_other-tutorials/01_using-hugging-face-integrations.md",
"level": 2,
"title": "Hosting your Gradio demos on Spaces",
"content": "[Hugging Face Spaces](https://hf.co/spaces) allows anyone to host their Gradio demos freely, and uploading your Gradio demos take a couple of minutes. You can head to [hf.co/new-space](https://huggingface.co/new-space), select the Gradio SDK, create an `app.py` file, and voila! You have a demo you can share with anyone else. To learn more, read [this guide how to host on Hugging Face Spaces using the website](https://huggingface.co/blog/gradio-spaces).\n\nAlternatively, you can create a Space programmatically, making use of the [huggingface_hub client library](https://huggingface.co/docs/huggingface_hub/index) library. Here's an example:\n\n```python\nfrom huggingface_hub import (\n create_repo,\n get_full_repo_name,\n upload_file,\n)\ncreate_repo(name=target_space_name, token=hf_token, repo_type=\"space\", space_sdk=\"gradio\")\nrepo_name = get_full_repo_name(model_id=target_space_name, token=hf_token)\nfile_url = upload_file(\n path_or_fileobj=\"file.txt\",\n path_in_repo=\"app.py\",\n repo_id=repo_name,\n repo_type=\"space\",\n token=hf_token,\n)\n```\n\nHere, `create_repo` creates a gradio repo with the target name under a specific account using that account's Write Token. `repo_name` gets the full repo name of the related repo. Finally `upload_file` uploads a file inside the repo with the name `app.py`."
},
{
"id": 203,
"parent": 199,
"path": "10_other-tutorials/01_using-hugging-face-integrations.md",
"level": 2,
"title": "Loading demos from Spaces",
"content": "You can also use and remix existing Gradio demos on Hugging Face Spaces. For example, you could take two existing Gradio demos on Spaces and put them as separate tabs and create a new demo. You can run this new demo locally, or upload it to Spaces, allowing endless possibilities to remix and create new demos!\n\nHere's an example that does exactly that:\n\n```python\nimport gradio as gr\n\nwith gr.Blocks() as demo:\n with gr.Tab(\"Translate to Spanish\"):\n gr.load(\"gradio/en2es\", src=\"spaces\")\n with gr.Tab(\"Translate to French\"):\n gr.load(\"abidlabs/en2fr\", src=\"spaces\")\n\ndemo.launch()\n```\n\nNotice that we use `gr.load()`, the same method we used to load models using Inference Endpoints. However, here we specify that the `src` is `spaces` (Hugging Face Spaces). \n\nNote: loading a Space in this way may result in slight differences from the original Space. In particular, any attributes that apply to the entire Blocks, such as the theme or custom CSS/JS, will not be loaded. You can copy these properties from the Space you are loading into your own `Blocks` object."
},
{
"id": 204,
"parent": 199,
"path": "10_other-tutorials/01_using-hugging-face-integrations.md",
"level": 2,
"title": "Demos with the `Pipeline` in `transformers`",
"content": "Hugging Face's popular `transformers` library has a very easy-to-use abstraction, [`pipeline()`](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/pipelines#transformers.pipeline) that handles most of the complex code to offer a simple API for common tasks. By specifying the task and an (optional) model, you can build a demo around an existing model with few lines of Python:\n\n```python\nimport gradio as gr\n\nfrom transformers import pipeline\n\npipe = pipeline(\"translation\", model=\"Helsinki-NLP/opus-mt-en-es\")\n\ndef predict(text):\n return pipe(text)[0][\"translation_text\"]\n\ndemo = gr.Interface(\n fn=predict,\n inputs='text',\n outputs='text',\n)\n\ndemo.launch()\n```\n\nBut `gradio` actually makes it even easier to convert a `pipeline` to a demo, simply by using the `gradio.Interface.from_pipeline` methods, which skips the need to specify the input and output components:\n\n```python\nfrom transformers import pipeline\nimport gradio as gr\n\npipe = pipeline(\"translation\", model=\"Helsinki-NLP/opus-mt-en-es\")\n\ndemo = gr.Interface.from_pipeline(pipe)\ndemo.launch()\n```\n\nThe previous code produces the following interface, which you can try right here in your browser:\n\n\"\n + html\n + \"
\"\n )\n pos_count = {\n \"char_count\": len(text),\n \"token_count\": 0,\n }\n pos_tokens = []\n\n for token in doc:\n pos_tokens.extend([(token.text, token.pos_), (\" \", None)])\n\n return pos_tokens, pos_count, html\n\ndemo = gr.Interface(\n text_analysis,\n gr.Textbox(placeholder=\"Enter sentence here...\"),\n [\"highlight\", \"json\", \"html\"],\n examples=[\n [\"What a beautiful morning for a walk!\"],\n [\"It was the best of times, it was the worst of times.\"],\n ],\n)\n\ndemo.launch()\n\n```\n$demo_text_analysis\n\n---\n\nAnd you're done! That's all you need to know to build a web-based GUI for your NER model.\n\nFun tip: you can share your NER demo instantly with others simply by setting `share=True` in `launch()`."
},
{
"id": 211,
"parent": null,
"path": "10_other-tutorials/wrapping-layouts.md",
"level": 1,
"title": "Wrapping Layouts",
"content": "Tags: LAYOUTS"
},
{
"id": 212,
"parent": 211,
"path": "10_other-tutorials/wrapping-layouts.md",
"level": 2,
"title": "Introduction",
"content": "Gradio features [blocks](https://www.gradio.app/docs/blocks) to easily layout applications. To use this feature, you need to stack or nest layout components and create a hierarchy with them. This isn't difficult to implement and maintain for small projects, but after the project gets more complex, this component hierarchy becomes difficult to maintain and reuse.\n\nIn this guide, we are going to explore how we can wrap the layout classes to create more maintainable and easy-to-read applications without sacrificing flexibility."
},
{
"id": 213,
"parent": 211,
"path": "10_other-tutorials/wrapping-layouts.md",
"level": 2,
"title": "Example",
"content": "We are going to follow the implementation from this Huggingface Space example:\n\n| product_id | int8 |
| inventory_count | int8 |
| price | float8 |
| product_name | varchar |
Price: $%{customdata[1]}'\n ))\n\nfig.update_layout(\n mapbox_style=\"open-street-map\",\n hovermode='closest',\n mapbox=dict(\n bearing=0,\n center=go.layout.mapbox.Center(\n lat=40.67,\n lon=-73.90\n ),\n pitch=0,\n zoom=9\n ),\n)\n```\n\nAbove, we create a scatter plot on mapbox by passing it our list of latitudes and longitudes to plot markers. We also pass in our custom data of names and prices for additional info to appear on every marker we hover over. Next we use `update_layout` to specify other map settings such as zoom, and centering.\n\nMore info [here](https://plotly.com/python/scattermapbox/) on scatter plots using Mapbox and Plotly." }, { "id": 294, "parent": 289, "path": "10_other-tutorials/plot-component-for-maps.md", "level": 2, "title": "Step 3 - Gradio App ⚡️", "content": "We will use two `gr.Number` components and a `gr.CheckboxGroup` to allow users of our app to specify price ranges and borough locations. We will then use the `gr.Plot` component as an output for our Plotly + Mapbox map we created earlier.\n\n```python\nwith gr.Blocks() as demo:\n with gr.Column():\n with gr.Row():\n min_price = gr.Number(value=250, label=\"Minimum Price\")\n max_price = gr.Number(value=1000, label=\"Maximum Price\")\n boroughs = gr.CheckboxGroup(choices=[\"Queens\", \"Brooklyn\", \"Manhattan\", \"Bronx\", \"Staten Island\"], value=[\"Queens\", \"Brooklyn\"], label=\"Select Boroughs:\")\n btn = gr.Button(value=\"Update Filter\")\n map = gr.Plot()\n demo.load(filter_map, [min_price, max_price, boroughs], map)\n btn.click(filter_map, [min_price, max_price, boroughs], map)\n```\n\nWe layout these components using the `gr.Column` and `gr.Row` and we'll also add event triggers for when the demo first loads and when our \"Update Filter\" button is clicked in order to trigger the map to update with our new filters.\n\nThis is what the full demo code looks like:\n\n```py\n# type: ignore\nimport gradio as gr\nimport plotly.graph_objects as go\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"gradio/NYC-Airbnb-Open-Data\", split=\"train\")\ndf = dataset.to_pandas()\n\ndef filter_map(min_price, max_price, boroughs):\n\n filtered_df = df[(df['neighbourhood_group'].isin(boroughs)) &\n (df['price'] > min_price) & (df['price'] < max_price)]\n names = filtered_df[\"name\"].tolist()\n prices = filtered_df[\"price\"].tolist()\n text_list = [(names[i], prices[i]) for i in range(0, len(names))]\n fig = go.Figure(go.Scattermapbox(\n customdata=text_list,\n lat=filtered_df['latitude'].tolist(),\n lon=filtered_df['longitude'].tolist(),\n mode='markers',\n marker=go.scattermapbox.Marker(\n size=6\n ),\n hoverinfo=\"text\",\n hovertemplate='Name: %{customdata[0]}
Price: $%{customdata[1]}'\n ))\n\n fig.update_layout(\n mapbox_style=\"open-street-map\",\n hovermode='closest',\n mapbox=dict(\n bearing=0,\n center=go.layout.mapbox.Center(\n lat=40.67,\n lon=-73.90\n ),\n pitch=0,\n zoom=9\n ),\n )\n\n return fig\n\nwith gr.Blocks() as demo:\n with gr.Column():\n with gr.Row():\n min_price = gr.Number(value=250, label=\"Minimum Price\")\n max_price = gr.Number(value=1000, label=\"Maximum Price\")\n boroughs = gr.CheckboxGroup(choices=[\"Queens\", \"Brooklyn\", \"Manhattan\", \"Bronx\", \"Staten Island\"], value=[\"Queens\", \"Brooklyn\"], label=\"Select Boroughs:\")\n btn = gr.Button(value=\"Update Filter\")\n map = gr.Plot()\n demo.load(filter_map, [min_price, max_price, boroughs], map)\n btn.click(filter_map, [min_price, max_price, boroughs], map)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```" }, { "id": 295, "parent": 289, "path": "10_other-tutorials/plot-component-for-maps.md", "level": 2, "title": "Step 4 - Deployment 🤗", "content": "If you run the code above, your app will start running locally.\nYou can even get a temporary shareable link by passing the `share=True` parameter to `launch`.\n\nBut what if you want to a permanent deployment solution?\nLet's deploy our Gradio app to the free HuggingFace Spaces platform.\n\nIf you haven't used Spaces before, follow the previous guide [here](/using_hugging_face_integrations)." }, { "id": 296, "parent": 289, "path": "10_other-tutorials/plot-component-for-maps.md", "level": 2, "title": "Conclusion 🎉", "content": "And you're all done! That's all the code you need to build a map demo.\n\nHere's a link to the demo [Map demo](https://huggingface.co/spaces/gradio/map_airbnb) and [complete code](https://huggingface.co/spaces/gradio/map_airbnb/blob/main/run.py) (on Hugging Face Spaces)" }, { "id": 297, "parent": null, "path": "10_other-tutorials/image-classification-with-vision-transformers.md", "level": 1, "title": "Image Classification with Vision Transformers", "content": "Related spaces: https://huggingface.co/spaces/abidlabs/vision-transformer\nTags: VISION, TRANSFORMERS, HUB" }, { "id": 298, "parent": 297, "path": "10_other-tutorials/image-classification-with-vision-transformers.md", "level": 2, "title": "Introduction", "content": "Image classification is a central task in computer vision. Building better classifiers to classify what object is present in a picture is an active area of research, as it has applications stretching from facial recognition to manufacturing quality control.\n\nState-of-the-art image classifiers are based on the _transformers_ architectures, originally popularized for NLP tasks. Such architectures are typically called vision transformers (ViT). Such models are perfect to use with Gradio's _image_ input component, so in this tutorial we will build a web demo to classify images using Gradio. We will be able to build the whole web application in a **single line of Python**, and it will look like the demo on the bottom of the page.\n\nLet's get started!" }, { "id": 299, "parent": 298, "path": "10_other-tutorials/image-classification-with-vision-transformers.md", "level": 3, "title": "Prerequisites", "content": "Make sure you have the `gradio` Python package already [installed](/getting_started)." }, { "id": 300, "parent": 297, "path": "10_other-tutorials/image-classification-with-vision-transformers.md", "level": 2, "title": "Step 1 — Choosing a Vision Image Classification Model", "content": "First, we will need an image classification model. For this tutorial, we will use a model from the [Hugging Face Model Hub](https://huggingface.co/models?pipeline_tag=image-classification). The Hub contains thousands of models covering dozens of different machine learning tasks.\n\nExpand the Tasks category on the left sidebar and select \"Image Classification\" as our task of interest. You will then see all of the models on the Hub that are designed to classify images.\n\nAt the time of writing, the most popular one is `google/vit-base-patch16-224`, which has been trained on ImageNet images at a resolution of 224x224 pixels. We will use this model for our demo." }, { "id": 301, "parent": 297, "path": "10_other-tutorials/image-classification-with-vision-transformers.md", "level": 2, "title": "Step 2 — Loading the Vision Transformer Model with Gradio", "content": "When using a model from the Hugging Face Hub, we do not need to define the input or output components for the demo. Similarly, we do not need to be concerned with the details of preprocessing or postprocessing.\nAll of these are automatically inferred from the model tags.\n\nBesides the import statement, it only takes a single line of Python to load and launch the demo.\n\nWe use the `gr.Interface.load()` method and pass in the path to the model including the `huggingface/` to designate that it is from the Hugging Face Hub.\n\n```python\nimport gradio as gr\n\ngr.Interface.load(\n \"huggingface/google/vit-base-patch16-224\",\n examples=[\"alligator.jpg\", \"laptop.jpg\"]).launch()\n```\n\nNotice that we have added one more parameter, the `examples`, which allows us to prepopulate our interfaces with a few predefined examples.\n\nThis produces the following interface, which you can try right here in your browser. When you input an image, it is automatically preprocessed and sent to the Hugging Face Hub API, where it is passed through the model and returned as a human-interpretable prediction. Try uploading your own image!\n\n
![\"Screen](\"https://user-images.githubusercontent.com/81195143/182252755-4a0e1ca8-fd25-40ff-8c91-c9da38aaa9ec.png\")
"
},
{
"id": 305,
"parent": 302,
"path": "10_other-tutorials/Gradio-and-Wandb-Integration.md",
"level": 2,
"title": "What are Hugging Face Spaces & Gradio?",
"content": ""
},
{
"id": 306,
"parent": 305,
"path": "10_other-tutorials/Gradio-and-Wandb-Integration.md",
"level": 3,
"title": "Gradio",
"content": "Gradio lets users demo their machine learning models as a web app, all in a few lines of Python. Gradio wraps any Python function (such as a machine learning model's inference function) into a user interface and the demos can be launched inside jupyter notebooks, colab notebooks, as well as embedded in your own website and hosted on Hugging Face Spaces for free.\n\nGet started [here](https://gradio.app/getting_started)"
},
{
"id": 307,
"parent": 305,
"path": "10_other-tutorials/Gradio-and-Wandb-Integration.md",
"level": 3,
"title": "Hugging Face Spaces",
"content": "Hugging Face Spaces is a free hosting option for Gradio demos. Spaces comes with 3 SDK options: Gradio, Streamlit and Static HTML demos. Spaces can be public or private and the workflow is similar to github repos. There are over 2000+ spaces currently on Hugging Face. Learn more about spaces [here](https://huggingface.co/spaces/launch)."
},
{
"id": 308,
"parent": 302,
"path": "10_other-tutorials/Gradio-and-Wandb-Integration.md",
"level": 2,
"title": "Setting up a Gradio Demo for JoJoGAN",
"content": "Now, let's walk you through how to do this on your own. We'll make the assumption that you're new to W&B and Gradio for the purposes of this tutorial.\n\nLet's get started!\n\n1. Create a W&B account\n\n Follow [these quick instructions](https://app.wandb.ai/login) to create your free account if you don’t have one already. It shouldn't take more than a couple minutes. Once you're done (or if you've already got an account), next, we'll run a quick colab.\n\n2. Open Colab Install Gradio and W&B\n\n We'll be following along with the colab provided in the JoJoGAN repo with some minor modifications to use Wandb and Gradio more effectively.\n\n [](https://colab.research.google.com/github/mchong6/JoJoGAN/blob/main/stylize.ipynb)\n\n Install Gradio and Wandb at the top:\n\n ```sh\n pip install gradio wandb\n ```\n\n3. Finetune StyleGAN and W&B experiment tracking\n\n This next step will open a W&B dashboard to track your experiments and a gradio panel showing pretrained models to choose from a drop down menu from a Gradio Demo hosted on Huggingface Spaces. Here's the code you need for that:\n\n ```python\n alpha = 1.0\n alpha = 1-alpha\n\n preserve_color = True\n num_iter = 100\n log_interval = 50\n\n samples = []\n column_names = [\"Reference (y)\", \"Style Code(w)\", \"Real Face Image(x)\"]\n\n wandb.init(project=\"JoJoGAN\")\n config = wandb.config\n config.num_iter = num_iter\n config.preserve_color = preserve_color\n wandb.log(\n {\"Style reference\": [wandb.Image(transforms.ToPILImage()(target_im))]},\n step=0)\n\n # load discriminator for perceptual loss\n discriminator = Discriminator(1024, 2).eval().to(device)\n ckpt = torch.load('models/stylegan2-ffhq-config-f.pt', map_location=lambda storage, loc: storage)\n discriminator.load_state_dict(ckpt[\"d\"], strict=False)\n\n # reset generator\n del generator\n generator = deepcopy(original_generator)\n\n g_optim = optim.Adam(generator.parameters(), lr=2e-3, betas=(0, 0.99))\n\n # Which layers to swap for generating a family of plausible real images -> fake image\n if preserve_color:\n id_swap = [9,11,15,16,17]\n else:\n id_swap = list(range(7, generator.n_latent))\n\n for idx in tqdm(range(num_iter)):\n mean_w = generator.get_latent(torch.randn([latents.size(0), latent_dim]).to(device)).unsqueeze(1).repeat(1, generator.n_latent, 1)\n in_latent = latents.clone()\n in_latent[:, id_swap] = alpha*latents[:, id_swap] + (1-alpha)*mean_w[:, id_swap]\n\n img = generator(in_latent, input_is_latent=True)\n\n with torch.no_grad():\n real_feat = discriminator(targets)\n fake_feat = discriminator(img)\n\n loss = sum([F.l1_loss(a, b) for a, b in zip(fake_feat, real_feat)])/len(fake_feat)\n\n wandb.log({\"loss\": loss}, step=idx)\n if idx % log_interval == 0:\n generator.eval()\n my_sample = generator(my_w, input_is_latent=True)\n generator.train()\n my_sample = transforms.ToPILImage()(utils.make_grid(my_sample, normalize=True, range=(-1, 1)))\n wandb.log(\n {\"Current stylization\": [wandb.Image(my_sample)]},\n step=idx)\n table_data = [\n wandb.Image(transforms.ToPILImage()(target_im)),\n wandb.Image(img),\n wandb.Image(my_sample),\n ]\n samples.append(table_data)\n\n g_optim.zero_grad()\n loss.backward()\n g_optim.step()\n\n out_table = wandb.Table(data=samples, columns=column_names)\n wandb.log({\"Current Samples\": out_table})\n ```\n4. Save, Download, and Load Model\n\n Here's how to save and download your model.\n\n ```python\n from PIL import Image\n import torch\n torch.backends.cudnn.benchmark = True\n from torchvision import transforms, utils\n from util import *\n import math\n import random\n import numpy as np\n from torch import nn, autograd, optim\n from torch.nn import functional as F\n from tqdm import tqdm\n import lpips\n from model import *\n from e4e_projection import projection as e4e_projection\n \n from copy import deepcopy\n import imageio\n \n import os\n import sys\n import torchvision.transforms as transforms\n from argparse import Namespace\n from e4e.models.psp import pSp\n from util import *\n from huggingface_hub import hf_hub_download\n from google.colab import files\n \n torch.save({\"g\": generator.state_dict()}, \"your-model-name.pt\")\n \n files.download('your-model-name.pt')\n \n latent_dim = 512\n device=\"cuda\"\n model_path_s = hf_hub_download(repo_id=\"akhaliq/jojogan-stylegan2-ffhq-config-f\", filename=\"stylegan2-ffhq-config-f.pt\")\n original_generator = Generator(1024, latent_dim, 8, 2).to(device)\n ckpt = torch.load(model_path_s, map_location=lambda storage, loc: storage)\n original_generator.load_state_dict(ckpt[\"g_ema\"], strict=False)\n mean_latent = original_generator.mean_latent(10000)\n \n generator = deepcopy(original_generator)\n \n ckpt = torch.load(\"/content/JoJoGAN/your-model-name.pt\", map_location=lambda storage, loc: storage)\n generator.load_state_dict(ckpt[\"g\"], strict=False)\n generator.eval()\n \n plt.rcParams['figure.dpi'] = 150\n \n transform = transforms.Compose(\n [\n transforms.Resize((1024, 1024)),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n ]\n )\n \n def inference(img):\n img.save('out.jpg')\n aligned_face = align_face('out.jpg')\n \n my_w = e4e_projection(aligned_face, \"out.pt\", device).unsqueeze(0)\n with torch.no_grad():\n my_sample = generator(my_w, input_is_latent=True)\n \n npimage = my_sample[0].cpu().permute(1, 2, 0).detach().numpy()\n imageio.imwrite('filename.jpeg', npimage)\n return 'filename.jpeg'\n ````\n\n5. Build a Gradio Demo\n\n ```python\n import gradio as gr\n \n title = \"JoJoGAN\"\n description = \"Gradio Demo for JoJoGAN: One Shot Face Stylization. To use it, simply upload your image, or click one of the examples to load them. Read more at the links below.\"\n \n demo = gr.Interface(\n inference,\n gr.Image(type=\"pil\"),\n gr.Image(type=\"file\"),\n title=title,\n description=description\n )\n \n demo.launch(share=True)\n ```\n\n6. Integrate Gradio into your W&B Dashboard\n\n The last step—integrating your Gradio demo with your W&B dashboard—is just one extra line:\n\n ```python\n demo.integrate(wandb=wandb)\n ```\n\n Once you call integrate, a demo will be created and you can integrate it into your dashboard or report.\n\n Outside of W&B with Web components, using the `gradio-app` tags, anyone can embed Gradio demos on HF spaces directly into their blogs, websites, documentation, etc.:\n \n ```html\n ![\"adding-panels\"](\"https://user-images.githubusercontent.com/7529846/214329314-70a3ff3d-27fb-408c-a4d1-4b58892a3854.jpeg\")
\n\nNext, search for Gradio Panel Extended in the Public Panels section and click `Add`.\n\n![\"gradio-panel-extended\"](\"https://user-images.githubusercontent.com/7529846/214325577-43226119-0292-46be-a62a-0c7a80646ebb.png\")
\n\nOnce you have added your Panel, click `Edit` to access to the Panel Options page and paste in the URL of your Gradio application.\n\n\n\n![\"Edit-Gradio-Panel-URL\"](\"https://user-images.githubusercontent.com/7529846/214334843-870fe726-0aa1-4b21-bbc6-0c48f56c48d8.png\")
"
},
{
"id": 339,
"parent": 337,
"path": "10_other-tutorials/Gradio-and-Comet.md",
"level": 2,
"title": "3. Embedding Hugging Face Spaces directly into your Comet Projects",
"content": "\n\nYou can also embed Gradio Applications that are hosted on Hugging Faces Spaces into your Comet Projects using the Hugging Face Spaces Panel.\n\nGo to your Comet Project page, and head over to the Panels tab. Click the `+ Add` button to bring up the Panels search page. Next, search for the Hugging Face Spaces Panel in the Public Panels section and click `Add`.\n\n![\"huggingface-spaces-panel\"](\"https://user-images.githubusercontent.com/7529846/214325606-99aa3af3-b284-4026-b423-d3d238797e12.png\")
\n\nOnce you have added your Panel, click Edit to access to the Panel Options page and paste in the path of your Hugging Face Space e.g. `pytorch/ResNet`\n\n![\"Edit-HF-Space\"](\"https://user-images.githubusercontent.com/7529846/214335868-c6f25dee-13db-4388-bcf5-65194f850b02.png\")
"
},
{
"id": 340,
"parent": 337,
"path": "10_other-tutorials/Gradio-and-Comet.md",
"level": 2,
"title": "4. Logging Model Inferences to Comet",
"content": "\n\n[](https://colab.research.google.com/github/comet-ml/comet-examples/blob/master/integrations/model-evaluation/gradio/notebooks/Logging_Model_Inferences_with_Comet_and_Gradio.ipynb)\n\nIn the previous examples, we demonstrated the various ways in which you can interact with a Gradio application through the Comet UI. Additionally, you can also log model inferences, such as SHAP plots, from your Gradio application to Comet.\n\nIn the following snippet, we're going to log inferences from a Text Generation model. We can persist an Experiment across multiple inference calls using Gradio's [State](https://www.gradio.app/docs/#state) object. This will allow you to log multiple inferences from a model to a single Experiment.\n\n```python\nimport comet_ml\nimport gradio as gr\nimport shap\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nif torch.cuda.is_available():\n device = \"cuda\"\nelse:\n device = \"cpu\"\n\nMODEL_NAME = \"gpt2\"\n\nmodel = AutoModelForCausalLM.from_pretrained(MODEL_NAME)"
},
{
"id": 341,
"parent": null,
"path": "10_other-tutorials/Gradio-and-Comet.md",
"level": 1,
"title": "set model decoder to true",
"content": "model.config.is_decoder = True"
},
{
"id": 342,
"parent": null,
"path": "10_other-tutorials/Gradio-and-Comet.md",
"level": 1,
"title": "set text-generation params under task_specific_params",
"content": "model.config.task_specific_params[\"text-generation\"] = {\n \"do_sample\": True,\n \"max_length\": 50,\n \"temperature\": 0.7,\n \"top_k\": 50,\n \"no_repeat_ngram_size\": 2,\n}\nmodel = model.to(device)\n\ntokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\nexplainer = shap.Explainer(model, tokenizer)\n\n\ndef start_experiment():\n \"\"\"Returns an APIExperiment object that is thread safe\n and can be used to log inferences to a single Experiment\n \"\"\"\n try:\n api = comet_ml.API()\n workspace = api.get_default_workspace()\n project_name = comet_ml.config.get_config()[\"comet.project_name\"]\n\n experiment = comet_ml.APIExperiment(\n workspace=workspace, project_name=project_name\n )\n experiment.log_other(\"Created from\", \"gradio-inference\")\n\n message = f\"Started Experiment: [{experiment.name}]({experiment.url})\"\n\n return (experiment, message)\n\n except Exception as e:\n return None, None\n\n\ndef predict(text, state, message):\n experiment = state\n\n shap_values = explainer([text])\n plot = shap.plots.text(shap_values, display=False)\n\n if experiment is not None:\n experiment.log_other(\"message\", message)\n experiment.log_html(plot)\n\n return plot\n\n\nwith gr.Blocks() as demo:\n start_experiment_btn = gr.Button(\"Start New Experiment\")\n experiment_status = gr.Markdown()\n\n # Log a message to the Experiment to provide more context\n experiment_message = gr.Textbox(label=\"Experiment Message\")\n experiment = gr.State()\n\n input_text = gr.Textbox(label=\"Input Text\", lines=5, interactive=True)\n submit_btn = gr.Button(\"Submit\")\n\n output = gr.HTML(interactive=True)\n\n start_experiment_btn.click(\n start_experiment, outputs=[experiment, experiment_status]\n )\n submit_btn.click(\n predict, inputs=[input_text, experiment, experiment_message], outputs=[output]\n )\n```\n\nInferences from this snippet will be saved in the HTML tab of your experiment.\n\n"
},
{
"id": 343,
"parent": 342,
"path": "10_other-tutorials/Gradio-and-Comet.md",
"level": 2,
"title": "Conclusion",
"content": "We hope you found this guide useful and that it provides some inspiration to help you build awesome model evaluation workflows with Comet and Gradio."
},
{
"id": 344,
"parent": 342,
"path": "10_other-tutorials/Gradio-and-Comet.md",
"level": 2,
"title": "How to contribute Gradio demos on HF spaces on the Comet organization",
"content": "- Create an account on Hugging Face [here](https://huggingface.co/join).\n- Add Gradio Demo under your username, see this [course](https://huggingface.co/course/chapter9/4?fw=pt) for setting up Gradio Demo on Hugging Face.\n- Request to join the Comet organization [here](https://huggingface.co/Comet)."
},
{
"id": 345,
"parent": 342,
"path": "10_other-tutorials/Gradio-and-Comet.md",
"level": 2,
"title": "Additional Resources",
"content": "- [Comet Documentation](https://www.comet.com/docs/v2/?utm_source=gradio&utm_medium=referral&utm_campaign=gradio-integration&utm_content=gradio-docs)"
},
{
"id": 346,
"parent": null,
"path": "10_other-tutorials/using-gradio-for-tabular-workflows.md",
"level": 1,
"title": "Using Gradio for Tabular Data Science Workflows",
"content": "Related spaces: https://huggingface.co/spaces/scikit-learn/gradio-skops-integration, https://huggingface.co/spaces/scikit-learn/tabular-playground, https://huggingface.co/spaces/merve/gradio-analysis-dashboard"
},
{
"id": 347,
"parent": 346,
"path": "10_other-tutorials/using-gradio-for-tabular-workflows.md",
"level": 2,
"title": "Introduction",
"content": "Tabular data science is the most widely used domain of machine learning, with problems ranging from customer segmentation to churn prediction. Throughout various stages of the tabular data science workflow, communicating your work to stakeholders or clients can be cumbersome; which prevents data scientists from focusing on what matters, such as data analysis and model building. Data scientists can end up spending hours building a dashboard that takes in dataframe and returning plots, or returning a prediction or plot of clusters in a dataset. In this guide, we'll go through how to use `gradio` to improve your data science workflows. We will also talk about how to use `gradio` and [skops](https://skops.readthedocs.io/en/stable/) to build interfaces with only one line of code!"
},
{
"id": 348,
"parent": 347,
"path": "10_other-tutorials/using-gradio-for-tabular-workflows.md",
"level": 3,
"title": "Prerequisites",
"content": "Make sure you have the `gradio` Python package already [installed](/getting_started)."
},
{
"id": 349,
"parent": 346,
"path": "10_other-tutorials/using-gradio-for-tabular-workflows.md",
"level": 2,
"title": "Let's Create a Simple Interface!",
"content": "We will take a look at how we can create a simple UI that predicts failures based on product information.\n\n```python\nimport gradio as gr\nimport pandas as pd\nimport joblib\nimport datasets\n\n\ninputs = [gr.Dataframe(row_count = (2, \"dynamic\"), col_count=(4,\"dynamic\"), label=\"Input Data\", interactive=1)]\n\noutputs = [gr.Dataframe(row_count = (2, \"dynamic\"), col_count=(1, \"fixed\"), label=\"Predictions\", headers=[\"Failures\"])]\n\nmodel = joblib.load(\"model.pkl\")"
},
{
"id": 350,
"parent": null,
"path": "10_other-tutorials/using-gradio-for-tabular-workflows.md",
"level": 1,
"title": "we will give our dataframe as example",
"content": "df = datasets.load_dataset(\"merve/supersoaker-failures\")\ndf = df[\"train\"].to_pandas()\n\ndef infer(input_dataframe):\n return pd.DataFrame(model.predict(input_dataframe))\n\ngr.Interface(fn = infer, inputs = inputs, outputs = outputs, examples = [[df.head(2)]]).launch()\n```\n\nLet's break down above code.\n\n- `fn`: the inference function that takes input dataframe and returns predictions.\n- `inputs`: the component we take our input with. We define our input as dataframe with 2 rows and 4 columns, which initially will look like an empty dataframe with the aforementioned shape. When the `row_count` is set to `dynamic`, you don't have to rely on the dataset you're inputting to pre-defined component.\n- `outputs`: The dataframe component that stores outputs. This UI can take single or multiple samples to infer, and returns 0 or 1 for each sample in one column, so we give `row_count` as 2 and `col_count` as 1 above. `headers` is a list made of header names for dataframe.\n- `examples`: You can either pass the input by dragging and dropping a CSV file, or a pandas DataFrame through examples, which headers will be automatically taken by the interface.\n\nWe will now create an example for a minimal data visualization dashboard. You can find a more comprehensive version in the related Spaces.\n\n![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gradio-guides/gif-version.gif\")
\n\nIt just takes a few lines of Python to create your own demo, so let's get started 💫"
},
{
"id": 392,
"parent": 391,
"path": "01_getting-started/01_quickstart.md",
"level": 2,
"title": "Installation",
"content": "**Prerequisite**: Gradio requires [Python 3.10 or higher](https://www.python.org/downloads/).\n\n\nWe recommend installing Gradio using `pip`, which is included by default in Python. Run this in your terminal or command prompt:\n\n```bash\npip install --upgrade gradio\n```\n\n\nTip: It is best to install Gradio in a virtual environment. Detailed installation instructions for all common operating systems are provided here."
},
{
"id": 393,
"parent": 391,
"path": "01_getting-started/01_quickstart.md",
"level": 2,
"title": "Building Your First Demo",
"content": "You can run Gradio in your favorite code editor, Jupyter notebook, Google Colab, or anywhere else you write Python. Let's write your first Gradio app:\n\n\n```py\nimport gradio as gr\n\ndef greet(name, intensity):\n return \"Hello, \" + name + \"!\" * int(intensity)\n\ndemo = gr.Interface(\n fn=greet,\n inputs=[\"text\", \"slider\"],\n outputs=[\"text\"],\n)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n\n\nTip: We shorten the imported name from gradio to gr. This is a widely adopted convention for better readability of code. \n\nNow, run your code. If you've written the Python code in a file named `app.py`, then you would run `python app.py` from the terminal.\n\nThe demo below will open in a browser on [http://localhost:7860](http://localhost:7860) if running from a file. If you are running within a notebook, the demo will appear embedded within the notebook.\n\n$demo_hello_world_4\n\nType your name in the textbox on the left, drag the slider, and then press the Submit button. You should see a friendly greeting on the right.\n\nTip: When developing locally, you can run your Gradio app in hot reload mode, which automatically reloads the Gradio app whenever you make changes to the file. To do this, simply type in gradio before the name of the file instead of python. In the example above, you would type: `gradio app.py` in your terminal. Learn more in the Hot Reloading Guide.\n\n\n**Understanding the `Interface` Class**\n\nYou'll notice that in order to make your first demo, you created an instance of the `gr.Interface` class. The `Interface` class is designed to create demos for machine learning models which accept one or more inputs, and return one or more outputs. \n\nThe `Interface` class has three core arguments:\n\n- `fn`: the function to wrap a user interface (UI) around\n- `inputs`: the Gradio component(s) to use for the input. The number of components should match the number of arguments in your function.\n- `outputs`: the Gradio component(s) to use for the output. The number of components should match the number of return values from your function.\n\nThe `fn` argument is very flexible -- you can pass *any* Python function that you want to wrap with a UI. In the example above, we saw a relatively simple function, but the function could be anything from a music generator to a tax calculator to the prediction function of a pretrained machine learning model.\n\nThe `inputs` and `outputs` arguments take one or more Gradio components. As we'll see, Gradio includes more than [30 built-in components](https://www.gradio.app/docs/gradio/introduction) (such as the `gr.Textbox()`, `gr.Image()`, and `gr.HTML()` components) that are designed for machine learning applications. \n\nTip: For the `inputs` and `outputs` arguments, you can pass in the name of these components as a string (`\"textbox\"`) or an instance of the class (`gr.Textbox()`).\n\nIf your function accepts more than one argument, as is the case above, pass a list of input components to `inputs`, with each input component corresponding to one of the arguments of the function, in order. The same holds true if your function returns more than one value: simply pass in a list of components to `outputs`. This flexibility makes the `Interface` class a very powerful way to create demos.\n\nWe'll dive deeper into the `gr.Interface` on our series on [building Interfaces](https://www.gradio.app/main/guides/the-interface-class)."
},
{
"id": 394,
"parent": 391,
"path": "01_getting-started/01_quickstart.md",
"level": 2,
"title": "Sharing Your Demo",
"content": "What good is a beautiful demo if you can't share it? Gradio lets you easily share a machine learning demo without having to worry about the hassle of hosting on a web server. Simply set `share=True` in `launch()`, and a publicly accessible URL will be created for your demo. Let's revisit our example demo, but change the last line as follows:\n\n```python\nimport gradio as gr\n\ndef greet(name):\n return \"Hello \" + name + \"!\"\n\ndemo = gr.Interface(fn=greet, inputs=\"textbox\", outputs=\"textbox\")\n \ndemo.launch(share=True) # Share your demo with just 1 extra parameter 🚀\n```\n\nWhen you run this code, a public URL will be generated for your demo in a matter of seconds, something like:\n\n👉 `https://a23dsf231adb.gradio.live`\n\nNow, anyone around the world can try your Gradio demo from their browser, while the machine learning model and all computation continues to run locally on your computer.\n\nTo learn more about sharing your demo, read our dedicated guide on [sharing your Gradio application](https://www.gradio.app/guides/sharing-your-app)."
},
{
"id": 395,
"parent": 391,
"path": "01_getting-started/01_quickstart.md",
"level": 2,
"title": "An Overview of Gradio",
"content": "So far, we've been discussing the `Interface` class, which is a high-level class that lets to build demos quickly with Gradio. But what else does Gradio include?"
},
{
"id": 396,
"parent": 395,
"path": "01_getting-started/01_quickstart.md",
"level": 3,
"title": "Custom Demos with `gr.Blocks`",
"content": "Gradio offers a low-level approach for designing web apps with more customizable layouts and data flows with the `gr.Blocks` class. Blocks supports things like controlling where components appear on the page, handling multiple data flows and more complex interactions (e.g. outputs can serve as inputs to other functions), and updating properties/visibility of components based on user interaction — still all in Python. \n\nYou can build very custom and complex applications using `gr.Blocks()`. For example, the popular image generation [Automatic1111 Web UI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) is built using Gradio Blocks. We dive deeper into the `gr.Blocks` on our series on [building with Blocks](https://www.gradio.app/guides/blocks-and-event-listeners)."
},
{
"id": 397,
"parent": 395,
"path": "01_getting-started/01_quickstart.md",
"level": 3,
"title": "Chatbots with `gr.ChatInterface`",
"content": "Gradio includes another high-level class, `gr.ChatInterface`, which is specifically designed to create Chatbot UIs. Similar to `Interface`, you supply a function and Gradio creates a fully working Chatbot UI. If you're interested in creating a chatbot, you can jump straight to [our dedicated guide on `gr.ChatInterface`](https://www.gradio.app/guides/creating-a-chatbot-fast)."
},
{
"id": 398,
"parent": 395,
"path": "01_getting-started/01_quickstart.md",
"level": 3,
"title": "The Gradio Python & JavaScript Ecosystem",
"content": "That's the gist of the core `gradio` Python library, but Gradio is actually so much more! It's an entire ecosystem of Python and JavaScript libraries that let you build machine learning applications, or query them programmatically, in Python or JavaScript. Here are other related parts of the Gradio ecosystem:\n\n* [Gradio Python Client](https://www.gradio.app/guides/getting-started-with-the-python-client) (`gradio_client`): query any Gradio app programmatically in Python.\n* [Gradio JavaScript Client](https://www.gradio.app/guides/getting-started-with-the-js-client) (`@gradio/client`): query any Gradio app programmatically in JavaScript.\n* [Gradio-Lite](https://www.gradio.app/guides/gradio-lite) (`@gradio/lite`): write Gradio apps in Python that run entirely in the browser (no server needed!), thanks to Pyodide. \n* [Hugging Face Spaces](https://huggingface.co/spaces): the most popular place to host Gradio applications — for free!"
},
{
"id": 399,
"parent": 391,
"path": "01_getting-started/01_quickstart.md",
"level": 2,
"title": "What's Next?",
"content": "Keep learning about Gradio sequentially using the Gradio Guides, which include explanations as well as example code and embedded interactive demos. Next up: [let's dive deeper into the Interface class](https://www.gradio.app/guides/the-interface-class).\n\nOr, if you already know the basics and are looking for something specific, you can search the more [technical API documentation](https://www.gradio.app/docs/)."
},
{
"id": 400,
"parent": null,
"path": "09_gradio-clients-and-lite/05_gradio-lite.md",
"level": 1,
"title": "Gradio-Lite: Serverless Gradio Running Entirely in Your Browser",
"content": "Tags: SERVERLESS, BROWSER, PYODIDE\n\nGradio is a popular Python library for creating interactive machine learning apps. Traditionally, Gradio applications have relied on server-side infrastructure to run, which can be a hurdle for developers who need to host their applications.\n\nEnter Gradio-lite (`@gradio/lite`): a library that leverages [Pyodide](https://pyodide.org/en/stable/) to bring Gradio directly to your browser. In this blog post, we'll explore what `@gradio/lite` is, go over example code, and discuss the benefits it offers for running Gradio applications."
},
{
"id": 401,
"parent": 400,
"path": "09_gradio-clients-and-lite/05_gradio-lite.md",
"level": 2,
"title": "What is `@gradio/lite`?",
"content": "`@gradio/lite` is a JavaScript library that enables you to run Gradio applications directly within your web browser. It achieves this by utilizing Pyodide, a Python runtime for WebAssembly, which allows Python code to be executed in the browser environment. With `@gradio/lite`, you can **write regular Python code for your Gradio applications**, and they will **run seamlessly in the browser** without the need for server-side infrastructure."
},
{
"id": 402,
"parent": 400,
"path": "09_gradio-clients-and-lite/05_gradio-lite.md",
"level": 2,
"title": "Getting Started",
"content": "Let's build a \"Hello World\" Gradio app in `@gradio/lite`"
},
{
"id": 403,
"parent": 402,
"path": "09_gradio-clients-and-lite/05_gradio-lite.md",
"level": 3,
"title": "1. Import JS and CSS",
"content": "Start by creating a new HTML file, if you don't have one already. Importing the JavaScript and CSS corresponding to the `@gradio/lite` package by using the following code:\n\n\n```html\n\n\t\n\t\t\n\t\t\n\t\n\n```\n\nNote that you should generally use the latest version of `@gradio/lite` that is available. You can see the [versions available here](https://www.jsdelivr.com/package/npm/@gradio/lite?tab=files)."
},
{
"id": 404,
"parent": 402,
"path": "09_gradio-clients-and-lite/05_gradio-lite.md",
"level": 3,
"title": "2. Create the `\n\n**Note**: If you use an audio pipeline, such as `automatic-speech-recognition`, you will need to put `transformers-js-py[audio]` in your `
![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gradio-guides/oauth_sign_in.png\")
\nAsk Me Anything\"),\n...\n```\n\nThe placeholder appears vertically and horizontally centered in the chatbot." }, { "id": 571, "parent": 566, "path": "05_chatbots/01_creating-a-chatbot-fast.md", "level": 2, "title": "Multimodal Chat Interface", "content": "You may want to add multimodal capabilities to your chat interface. For example, you may want users to be able to upload images or files to your chatbot and ask questions about them. You can make your chatbot \"multimodal\" by passing in a single parameter (`multimodal=True`) to the `gr.ChatInterface` class.\n\nWhen `multimodal=True`, the signature of your chat function changes slightly: the first parameter of your function (what we referred to as `message` above) should accept a dictionary consisting of the submitted text and uploaded files that looks like this: \n\n```py\n{\n \"text\": \"user input\", \n \"files\": [\n \"updated_file_1_path.ext\",\n \"updated_file_2_path.ext\", \n ...\n ]\n}\n```\n\nThis second parameter of your chat function, `history`, will be in the same openai-style dictionary format as before. However, if the history contains uploaded files, the `content` key for a file will be not a string, but rather a single-element tuple consisting of the filepath. Each file will be a separate message in the history. So after uploading two files and asking a question, your history might look like this:\n\n```python\n[\n {\"role\": \"user\", \"content\": (\"cat1.png\")},\n {\"role\": \"user\", \"content\": (\"cat2.png\")},\n {\"role\": \"user\", \"content\": \"What's the difference between these two images?\"},\n]\n```\n\nThe return type of your chat function does *not change* when setting `multimodal=True` (i.e. in the simplest case, you should still return a string value). We discuss more complex cases, e.g. returning files [below](#returning-complex-responses).\n\nIf you are customizing a multimodal chat interface, you should pass in an instance of `gr.MultimodalTextbox` to the `textbox` parameter. Here's an example that illustrates how to set up and customize and multimodal chat interface:\n \n\n```python\nimport gradio as gr\n\ndef count_images(message, history):\n num_images = len(message[\"files\"])\n total_images = 0\n for message in history:\n if isinstance(message[\"content\"], tuple):\n total_images += 1\n return f\"You just uploaded {num_images} images, total uploaded: {total_images+num_images}\"\n\ndemo = gr.ChatInterface(\n fn=count_images, \n type=\"messages\", \n examples=[\n {\"text\": \"No files\", \"files\": []}\n ], \n multimodal=True,\n textbox=gr.MultimodalTextbox(file_count=\"multiple\", file_types=[\"image\"])\n)\n\ndemo.launch()\n```" }, { "id": 572, "parent": 566, "path": "05_chatbots/01_creating-a-chatbot-fast.md", "level": 2, "title": "Additional Inputs", "content": "You may want to add additional inputs to your chat function and expose them to your users through the chat UI. For example, you could add a textbox for a system prompt, or a slider that sets the number of tokens in the chatbot's response. The `gr.ChatInterface` class supports an `additional_inputs` parameter which can be used to add additional input components.\n\nThe `additional_inputs` parameters accepts a component or a list of components. You can pass the component instances directly, or use their string shortcuts (e.g. `\"textbox\"` instead of `gr.Textbox()`). If you pass in component instances, and they have _not_ already been rendered, then the components will appear underneath the chatbot within a `gr.Accordion()`. \n\nHere's a complete example:\n\n```py\nimport gradio as gr\nimport time\n\ndef echo(message, history, system_prompt, tokens):\n response = f\"System prompt: {system_prompt}\\n Message: {message}.\"\n for i in range(min(len(response), int(tokens))):\n time.sleep(0.05)\n yield response[: i + 1]\n\ndemo = gr.ChatInterface(\n echo,\n type=\"messages\",\n additional_inputs=[\n gr.Textbox(\"You are helpful AI.\", label=\"System Prompt\"),\n gr.Slider(10, 100),\n ],\n)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n\nIf the components you pass into the `additional_inputs` have already been rendered in a parent `gr.Blocks()`, then they will _not_ be re-rendered in the accordion. This provides flexibility in deciding where to lay out the input components. In the example below, we position the `gr.Textbox()` on top of the Chatbot UI, while keeping the slider underneath.\n\n```python\nimport gradio as gr\nimport time\n\ndef echo(message, history, system_prompt, tokens):\n response = f\"System prompt: {system_prompt}\\n Message: {message}.\"\n for i in range(min(len(response), int(tokens))):\n time.sleep(0.05)\n yield response[: i+1]\n\nwith gr.Blocks() as demo:\n system_prompt = gr.Textbox(\"You are helpful AI.\", label=\"System Prompt\")\n slider = gr.Slider(10, 100, render=False)\n\n gr.ChatInterface(\n echo, additional_inputs=[system_prompt, slider], type=\"messages\"\n )\n\ndemo.launch()\n```\n\n**Examples with additional inputs**\n\nYou can also add example values for your additional inputs. Pass in a list of lists to the `examples` parameter, where each inner list represents one sample, and each inner list should be `1 + len(additional_inputs)` long. The first element in the inner list should be the example value for the chat message, and each subsequent element should be an example value for one of the additional inputs, in order. When additional inputs are provided, examples are rendered in a table underneath the chat interface.\n\nIf you need to create something even more custom, then its best to construct the chatbot UI using the low-level `gr.Blocks()` API. We have [a dedicated guide for that here](/guides/creating-a-custom-chatbot-with-blocks)." }, { "id": 573, "parent": 566, "path": "05_chatbots/01_creating-a-chatbot-fast.md", "level": 2, "title": "Returning Complex Responses", "content": "We mentioned earlier that in the simplest case, your chat function should return a `str` response, which will be rendered as text in the chatbot. However, you can also return more complex responses as we discuss below:\n\n**Returning Gradio components**\n\nCurrently, the following Gradio components can be displayed inside the chat interface:\n* `gr.Image`\n* `gr.Plot`\n* `gr.Audio`\n* `gr.HTML`\n* `gr.Video`\n* `gr.Gallery`\n\nSimply return one of these components from your function to use it with `gr.ChatInterface`. Here's an example:\n\n```py\nimport gradio as gr\n\ndef music(message, history):\n if message.strip():\n return gr.Audio(\"https://github.com/gradio-app/gradio/raw/main/test/test_files/audio_sample.wav\")\n else:\n return \"Please provide the name of an artist\"\n\ngr.ChatInterface(\n fake,\n type=\"messages\",\n textbox=gr.Textbox(placeholder=\"Which artist's music do you want to listen to?\", scale=7),\n).launch()\n```\n\n\n**Returning image, audio, video, or other files**:\n\nSometimes, you don't want to return a complete Gradio component, but rather simply an image/audio/video/other file to be displayed inside the chatbot. You can do this by returning a complete openai-style dictionary from your chat function. The dictionary should consist of the following keys:\n\n* `role`: set to `\"assistant\"`\n* `content`: set to a dictionary with key `path` and value the filepath or URL you'd like to return\n\nHere is an example:\n\n```py\nimport gradio as gr\n\ndef fake(message, history):\n if message.strip():\n return {\n \"role\": \"assistant\", \n \"content\": {\n \"path\": \"https://github.com/gradio-app/gradio/raw/main/test/test_files/audio_sample.wav\"\n }\n }\n else:\n return \"Please provide the name of an artist\"\n\ngr.ChatInterface(\n fake,\n type=\"messages\",\n textbox=gr.Textbox(placeholder=\"Which artist's music do you want to listen to?\", scale=7),\n chatbot=gr.Chatbot(placeholder=\"Play music by any artist!\"),\n).launch()\n```\n\n\n**Providing preset responses**\n\nYou may want to provide preset responses that a user can choose between when conversing with your chatbot. You can add the `options` key to the dictionary returned from your chat function to set these responses. The value corresponding to the `options` key should be a list of dictionaries, each with a `value` (a string that is the value that should be sent to the chat function when this response is clicked) and an optional `label` (if provided, is the text displayed as the preset response instead of the `value`). \n\nThis example illustrates how to use preset responses:\n\n```python\nimport gradio as gr\n\nexample_code = '''\nHere's the code I generated:\n\ndef greet(x):\n return f\"Hello, {x}!\"\n\nIs this correct?\n'''\n\ndef chat(message, history):\n if message == \"Yes, that's correct.\":\n return \"Great!\"\n else:\n return {\n \"role\": \"assistant\",\n \"content\": example_code,\n \"options\": [\n {\"value\": \"Yes, that's correct.\", \"label\": \"Yes\"},\n {\"value\": \"No\"}\n ]\n }\n\ndemo = gr.ChatInterface(\n chat,\n type=\"messages\",\n examples=[\"Write a Python function that takes a string and returns a greeting.\"]\n)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```" }, { "id": 574, "parent": 566, "path": "05_chatbots/01_creating-a-chatbot-fast.md", "level": 2, "title": "Using Your Chatbot via API", "content": "Once you've built your Gradio chat interface and are hosting it on [Hugging Face Spaces](https://hf.space) or somewhere else, then you can query it with a simple API at the `/chat` endpoint. The endpoint just expects the user's message (and potentially additional inputs if you have set any using the `additional_inputs` parameter), and will return the response, internally keeping track of the messages sent so far.\n\n[](https://github.com/gradio-app/gradio/assets/1778297/7b10d6db-6476-4e2e-bebd-ecda802c3b8f)\n\nTo use the endpoint, you should use either the [Gradio Python Client](/guides/getting-started-with-the-python-client) or the [Gradio JS client](/guides/getting-started-with-the-js-client). Or, you can deploy your Chat Interface to other platforms, such as [Discord](../guides/creating-a-discord-bot-from-a-gradio-app)." }, { "id": 575, "parent": 566, "path": "05_chatbots/01_creating-a-chatbot-fast.md", "level": 2, "title": "What's Next?", "content": "Now that you've learned about the `gr.ChatInterface` class and how it can be used to create chatbot UIs quickly, we recommend reading one of the following:\n\n* [Our next Guide](../guides/chatinterface-examples) shows examples of how to use `gr.ChatInterface` with popular LLM libraries.\n* If you'd like to build very custom chat applications from scratch, you can build them using the low-level Blocks API, as [discussed in this Guide](../guides/creating-a-custom-chatbot-with-blocks).\n* Once you've deployed your Gradio Chat Interface, its easy to use it other applications because of the built-in API. Here's a tutorial on [how to deploy a Gradio chat interface as a Discord bot](../guides/creating-a-discord-bot-from-a-gradio-app)." }, { "id": 576, "parent": null, "path": "05_chatbots/02_chatinterface-examples.md", "level": 1, "title": "Using Popular LLM libraries and APIs", "content": "Tags: LLM, CHATBOT, API\n\nIn this Guide, we go through several examples of how to use `gr.ChatInterface` with popular LLM libraries and API providers.\n\nWe will cover the following libraries and API providers:\n\n* [Llama Index](#llama-index)\n* [LangChain](#lang-chain)\n* [OpenAI](#open-ai)\n* [Hugging Face `transformers`](#hugging-face-transformers)\n* [SambaNova](#samba-nova)\n* [Hyperbolic](#hyperbolic)\n* [Anthropic's Claude](#anthropics-claude)\n\nFor many LLM libraries and providers, there exist community-maintained integration libraries that make it even easier to spin up Gradio apps. We reference these libraries in the appropriate sections below." }, { "id": 577, "parent": 576, "path": "05_chatbots/02_chatinterface-examples.md", "level": 2, "title": "Llama Index", "content": "Let's start by using `llama-index` on top of `openai` to build a RAG chatbot on any text or PDF files that you can demo and share in less than 30 lines of code. You'll need to have an OpenAI key for this example (keep reading for the free, open-source equivalent!)\n\n```py\n# This is a simple RAG chatbot built on top of Llama Index and Gradio. It allows you to upload any text or PDF files and ask questions about them!\n# Before running this, make sure you have exported your OpenAI API key as an environment variable:\n# export OPENAI_API_KEY=\"your-openai-api-key\"\n\nfrom llama_index.core import VectorStoreIndex, SimpleDirectoryReader\nimport gradio as gr\n\ndef answer(message, history):\n files = []\n for msg in history:\n if msg['role'] == \"user\" and isinstance(msg['content'], tuple):\n files.append(msg['content'][0])\n for file in message[\"files\"]:\n files.append(file)\n\n documents = SimpleDirectoryReader(input_files=files).load_data()\n index = VectorStoreIndex.from_documents(documents)\n query_engine = index.as_query_engine()\n return str(query_engine.query(message[\"text\"]))\n\ndemo = gr.ChatInterface(\n answer,\n type=\"messages\",\n title=\"Llama Index RAG Chatbot\",\n description=\"Upload any text or pdf files and ask questions about them!\",\n textbox=gr.MultimodalTextbox(file_types=[\".pdf\", \".txt\"]),\n multimodal=True\n)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```" }, { "id": 578, "parent": 576, "path": "05_chatbots/02_chatinterface-examples.md", "level": 2, "title": "LangChain", "content": "Here's an example using `langchain` on top of `openai` to build a general-purpose chatbot. As before, you'll need to have an OpenAI key for this example.\n\n```py\n# This is a simple general-purpose chatbot built on top of LangChain and Gradio.\n# Before running this, make sure you have exported your OpenAI API key as an environment variable:\n# export OPENAI_API_KEY=\"your-openai-api-key\"\n\nfrom langchain_openai import ChatOpenAI\nfrom langchain.schema import AIMessage, HumanMessage\nimport gradio as gr\n\nmodel = ChatOpenAI(model=\"gpt-4o-mini\")\n\ndef predict(message, history):\n history_langchain_format = []\n for msg in history:\n if msg['role'] == \"user\":\n history_langchain_format.append(HumanMessage(content=msg['content']))\n elif msg['role'] == \"assistant\":\n history_langchain_format.append(AIMessage(content=msg['content']))\n history_langchain_format.append(HumanMessage(content=message))\n gpt_response = model.invoke(history_langchain_format)\n return gpt_response.content\n\ndemo = gr.ChatInterface(\n predict,\n type=\"messages\"\n)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n\nTip: For quick prototyping, the community-maintained langchain-gradio repo makes it even easier to build chatbots on top of LangChain." }, { "id": 579, "parent": 576, "path": "05_chatbots/02_chatinterface-examples.md", "level": 2, "title": "OpenAI", "content": "Of course, we could also use the `openai` library directy. Here a similar example to the LangChain , but this time with streaming as well:\n\nTip: For quick prototyping, the openai-gradio library makes it even easier to build chatbots on top of OpenAI models." }, { "id": 580, "parent": 576, "path": "05_chatbots/02_chatinterface-examples.md", "level": 2, "title": "Hugging Face `transformers`", "content": "Of course, in many cases you want to run a chatbot locally. Here's the equivalent example using the SmolLM2-135M-Instruct model using the Hugging Face `transformers` library.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport gradio as gr\n\ncheckpoint = \"HuggingFaceTB/SmolLM2-135M-Instruct\"\ndevice = \"cpu\" # \"cuda\" or \"cpu\"\ntokenizer = AutoTokenizer.from_pretrained(checkpoint)\nmodel = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)\n\ndef predict(message, history):\n history.append({\"role\": \"user\", \"content\": message})\n input_text = tokenizer.apply_chat_template(history, tokenize=False)\n inputs = tokenizer.encode(input_text, return_tensors=\"pt\").to(device) # type: ignore\n outputs = model.generate(inputs, max_new_tokens=100, temperature=0.2, top_p=0.9, do_sample=True)\n decoded = tokenizer.decode(outputs[0])\n response = decoded.split(\"<|im_start|>assistant\\n\")[-1].split(\"<|im_end|>\")[0]\n return response\n\ndemo = gr.ChatInterface(predict, type=\"messages\")\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```" }, { "id": 581, "parent": 576, "path": "05_chatbots/02_chatinterface-examples.md", "level": 2, "title": "SambaNova", "content": "The SambaNova Cloud API provides access to full-precision open-source models, such as the Llama family. Here's an example of how to build a Gradio app around the SambaNova API\n\n```py\n# This is a simple general-purpose chatbot built on top of SambaNova API. \n# Before running this, make sure you have exported your SambaNova API key as an environment variable:\n# export SAMBANOVA_API_KEY=\"your-sambanova-api-key\"\n\nimport os\nimport gradio as gr\nfrom openai import OpenAI\n\napi_key = os.getenv(\"SAMBANOVA_API_KEY\")\n\nclient = OpenAI(\n base_url=\"https://api.sambanova.ai/v1/\",\n api_key=api_key,\n)\n\ndef predict(message, history):\n history.append({\"role\": \"user\", \"content\": message})\n stream = client.chat.completions.create(messages=history, model=\"Meta-Llama-3.1-70B-Instruct-8k\", stream=True)\n chunks = []\n for chunk in stream:\n chunks.append(chunk.choices[0].delta.content or \"\")\n yield \"\".join(chunks)\n\ndemo = gr.ChatInterface(predict, type=\"messages\")\n\nif __name__ == \"__main__\":\n demo.launch()\n\n\n```\n\nTip: For quick prototyping, the sambanova-gradio library makes it even easier to build chatbots on top of SambaNova models." }, { "id": 582, "parent": 576, "path": "05_chatbots/02_chatinterface-examples.md", "level": 2, "title": "Hyperbolic", "content": "The Hyperbolic AI API provides access to many open-source models, such as the Llama family. Here's an example of how to build a Gradio app around the SambaNova API\n\n```py\n# This is a simple general-purpose chatbot built on top of Hyperbolic API. \n# Before running this, make sure you have exported your Hyperbolic API key as an environment variable:\n# export HYPERBOLIC_API_KEY=\"your-hyperbolic-api-key\"\n\nimport os\nimport gradio as gr\nfrom openai import OpenAI\n\napi_key = os.getenv(\"HYPERBOLIC_API_KEY\")\n\nclient = OpenAI(\n base_url=\"https://api.hyperbolic.xyz/v1/\",\n api_key=api_key,\n)\n\ndef predict(message, history):\n history.append({\"role\": \"user\", \"content\": message})\n stream = client.chat.completions.create(messages=history, model=\"gpt-4o-mini\", stream=True)\n chunks = []\n for chunk in stream:\n chunks.append(chunk.choices[0].delta.content or \"\")\n yield \"\".join(chunks)\n\ndemo = gr.ChatInterface(predict, type=\"messages\")\n\nif __name__ == \"__main__\":\n demo.launch()\n\n\n```\n\nTip: For quick prototyping, the hyperbolic-gradio library makes it even easier to build chatbots on top of Hyperbolic models." }, { "id": 583, "parent": 576, "path": "05_chatbots/02_chatinterface-examples.md", "level": 2, "title": "Anthropic's Claude ", "content": "Anthropic's Claude model can also be used via API. Here's a simple 20 questions-style game built on top of the Anthropic API:\n\n```py\n# This is a simple 20 questions-style game built on top of the Anthropic API.\n# Before running this, make sure you have exported your Anthropic API key as an environment variable:\n# export ANTHROPIC_API_KEY=\"your-anthropic-api-key\"\n\nimport anthropic\nimport gradio as gr\n\nclient = anthropic.Anthropic()\n\ndef predict(message, history):\n keys_to_keep = [\"role\", \"content\"]\n history = [{k: d[k] for k in keys_to_keep if k in d} for d in history]\n history.append({\"role\": \"user\", \"content\": message})\n if len(history) > 20:\n history.append({\"role\": \"user\", \"content\": \"DONE\"})\n output = client.messages.create(\n messages=history, # type: ignore\n model=\"claude-3-5-sonnet-20241022\",\n max_tokens=1000,\n system=\"You are guessing an object that the user is thinking of. You can ask 10 yes/no questions. Keep asking questions until the user says DONE\"\n )\n return {\n \"role\": \"assistant\",\n \"content\": output.content[0].text, # type: ignore\n \"options\": [{\"value\": \"Yes\"}, {\"value\": \"No\"}]\n }\n\nplaceholder = \"\"\"\n
10 Questions
Think of a person, place, or thing. I'll ask you 10 yes/no questions to try and guess it.\n
![](\"https://gradio-builds.s3.amazonaws.com/demo-files/discordbots/guide/echo_slash.gif\")
"
},
{
"id": 594,
"parent": 587,
"path": "05_chatbots/06_creating-a-discord-bot-from-a-gradio-app.md",
"level": 3,
"title": "Using the `gradio_client.Client` Class",
"content": "You can also create a discord bot from a deployed gradio app with python.\n\n```python\nimport gradio_client as grc\ngrc.Client(\"freddyaboulton/echo-chatbot\").deploy_discord()\n```"
},
{
"id": 595,
"parent": 584,
"path": "05_chatbots/06_creating-a-discord-bot-from-a-gradio-app.md",
"level": 2,
"title": "🦾 Using State of The Art LLMs 🦾",
"content": "We have created an organization on Hugging Face called [gradio-discord-bots](https://huggingface.co/gradio-discord-bots) containing several template spaces that explain how to deploy state of the art LLMs powered by gradio as discord bots.\n\nThe easiest way to get started is by deploying Meta's Llama 2 LLM with 70 billion parameter. Simply go to this [space](https://huggingface.co/spaces/gradio-discord-bots/Llama-2-70b-chat-hf) and follow the instructions.\n\nThe deployment can be done in one line! 🤯\n\n```python\nimport gradio_client as grc\ngrc.Client(\"ysharma/Explore_llamav2_with_TGI\").deploy_discord(to_id=\"llama2-70b-discord-bot\")\n```"
},
{
"id": 596,
"parent": 584,
"path": "05_chatbots/06_creating-a-discord-bot-from-a-gradio-app.md",
"level": 2,
"title": "🦜 Additional LLMs 🦜",
"content": "In addition to Meta's 70 billion Llama 2 model, we have prepared template spaces for the following LLMs and deployment options:\n\n- [gpt-3.5-turbo](https://huggingface.co/spaces/gradio-discord-bots/gpt-35-turbo), powered by openai. Required OpenAI key.\n- [falcon-7b-instruct](https://huggingface.co/spaces/gradio-discord-bots/falcon-7b-instruct) powered by Hugging Face Inference Endpoints.\n- [Llama-2-13b-chat-hf](https://huggingface.co/spaces/gradio-discord-bots/Llama-2-13b-chat-hf) powered by Hugging Face Inference Endpoints.\n- [Llama-2-13b-chat-hf](https://huggingface.co/spaces/gradio-discord-bots/llama-2-13b-chat-transformers) powered by Hugging Face transformers.\n\nTo deploy any of these models to discord, simply follow the instructions in the linked space for that model."
},
{
"id": 597,
"parent": 584,
"path": "05_chatbots/06_creating-a-discord-bot-from-a-gradio-app.md",
"level": 2,
"title": "Deploying non-chat gradio apps to discord",
"content": "As mentioned above, you don't need a `gr.ChatInterface` if you want to deploy your gradio app to discord. All that's needed is an api route that takes in a single string and outputs a single string.\n\nThe following code will deploy a space that translates english to german as a discord bot.\n\n```python\nimport gradio_client as grc\nclient = grc.Client(\"freddyaboulton/english-to-german\")\nclient.deploy_discord(api_names=['german'])\n```"
},
{
"id": 598,
"parent": 584,
"path": "05_chatbots/06_creating-a-discord-bot-from-a-gradio-app.md",
"level": 2,
"title": "Conclusion",
"content": "That's it for this guide! We're really excited about this feature. Tag [@Gradio](https://twitter.com/Gradio) on twitter and show us how your discord community interacts with your discord bots."
},
{
"id": 599,
"parent": null,
"path": "06_data-science-and-plots/04_connecting-to-a-database.md",
"level": 1,
"title": "Connecting to a Database",
"content": "The data you wish to visualize may be stored in a database. Let's use SQLAlchemy to quickly extract database content into pandas Dataframe format so we can use it in gradio.\n\nFirst install `pip install sqlalchemy` and then let's see some examples."
},
{
"id": 600,
"parent": 599,
"path": "06_data-science-and-plots/04_connecting-to-a-database.md",
"level": 2,
"title": "SQLite",
"content": "```python\nfrom sqlalchemy import create_engine\nimport pandas as pd\n\nengine = create_engine('sqlite:///your_database.db')\n\nwith gr.Blocks() as demo:\n gr.LinePlot(pd.read_sql_query(\"SELECT time, price from flight_info;\", engine), x=\"time\", y=\"price\")\n```\n\nLet's see a a more interactive plot involving filters that modify your SQL query:\n\n```python\nfrom sqlalchemy import create_engine\nimport pandas as pd\n\nengine = create_engine('sqlite:///your_database.db')\n\nwith gr.Blocks() as demo:\n origin = gr.Dropdown([\"DFW\", \"DAL\", \"HOU\"], value=\"DFW\", label=\"Origin\")\n\n gr.LinePlot(lambda origin: pd.read_sql_query(f\"SELECT time, price from flight_info WHERE origin = {origin};\", engine), inputs=origin, x=\"time\", y=\"price\")\n```"
},
{
"id": 601,
"parent": 599,
"path": "06_data-science-and-plots/04_connecting-to-a-database.md",
"level": 2,
"title": "Postgres, mySQL, and other databases",
"content": "If you're using a different database format, all you have to do is swap out the engine, e.g.\n\n```python\nengine = create_engine('postgresql://username:password@host:port/database_name')\n```\n\n```python\nengine = create_engine('mysql://username:password@host:port/database_name')\n```\n\n```python\nengine = create_engine('oracle://username:password@host:port/database_name')\n```"
},
{
"id": 602,
"parent": null,
"path": "06_data-science-and-plots/03_filters-tables-and-stats.md",
"level": 1,
"title": "Filters, Tables and Stats",
"content": "Your dashboard will likely consist of more than just plots. Let's take a look at some of the other common components of a dashboard."
},
{
"id": 603,
"parent": 602,
"path": "06_data-science-and-plots/03_filters-tables-and-stats.md",
"level": 2,
"title": "Filters",
"content": "Use any of the standard Gradio form components to filter your data. You can do this via event listeners or function-as-value syntax. Let's look at the event listener approach first:\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n with gr.Row():\n origin = gr.Dropdown([\"All\", \"DFW\", \"DAL\", \"HOU\"], value=\"All\", label=\"Origin\")\n destination = gr.Dropdown([\"All\", \"JFK\", \"LGA\", \"EWR\"], value=\"All\", label=\"Destination\")\n max_price = gr.Slider(0, 1000, value=1000, label=\"Max Price\")\n\n plt = gr.ScatterPlot(df, x=\"time\", y=\"price\", inputs=[origin, destination, max_price])\n\n @gr.on(inputs=[origin, destination, max_price], outputs=plt)\n def filtered_data(origin, destination, max_price):\n _df = df[df[\"price\"] <= max_price]\n if origin != \"All\":\n _df = _df[_df[\"origin\"] == origin]\n if destination != \"All\":\n _df = _df[_df[\"destination\"] == destination]\n return _df\n\n \nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_filters_events\n\nAnd this would be the function-as-value approach for the same demo.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n with gr.Row():\n origin = gr.Dropdown([\"All\", \"DFW\", \"DAL\", \"HOU\"], value=\"All\", label=\"Origin\")\n destination = gr.Dropdown([\"All\", \"JFK\", \"LGA\", \"EWR\"], value=\"All\", label=\"Destination\")\n max_price = gr.Slider(0, 1000, value=1000, label=\"Max Price\")\n\n def filtered_data(origin, destination, max_price):\n _df = df[df[\"price\"] <= max_price]\n if origin != \"All\":\n _df = _df[_df[\"origin\"] == origin]\n if destination != \"All\":\n _df = _df[_df[\"destination\"] == destination]\n return _df\n\n gr.ScatterPlot(filtered_data, x=\"time\", y=\"price\", inputs=[origin, destination, max_price])\n \nif __name__ == \"__main__\":\n demo.launch()\n```"
},
{
"id": 604,
"parent": 602,
"path": "06_data-science-and-plots/03_filters-tables-and-stats.md",
"level": 2,
"title": "Tables and Stats",
"content": "Add `gr.DataFrame` and `gr.Label` to your dashboard for some hard numbers.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n with gr.Row():\n gr.Label(len(df), label=\"Flight Count\")\n gr.Label(f\"${df['price'].min()}\", label=\"Cheapest Flight\")\n gr.DataFrame(df)\n\n \nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_tables_stats"
},
{
"id": 605,
"parent": null,
"path": "06_data-science-and-plots/02_time-plots.md",
"level": 1,
"title": "Time Plots",
"content": "Creating visualizations with a time x-axis is a common use case. Let's dive in!"
},
{
"id": 606,
"parent": 605,
"path": "06_data-science-and-plots/02_time-plots.md",
"level": 2,
"title": "Creating a Plot with a pd.Dataframe",
"content": "Time plots need a datetime column on the x-axis. Here's a simple example with some flight data:\n\n```py\nimport gradio as gr\nimport pandas as pd\nimport numpy as np\nimport random\n\nfrom datetime import datetime, timedelta\nnow = datetime.now()\n\ndf = pd.DataFrame({\n 'time': [now - timedelta(minutes=5*i) for i in range(25)],\n 'price': np.random.randint(100, 1000, 25),\n 'origin': [random.choice([\"DFW\", \"DAL\", \"HOU\"]) for _ in range(25)],\n 'destination': [random.choice([\"JFK\", \"LGA\", \"EWR\"]) for _ in range(25)],\n})\n\nwith gr.Blocks() as demo:\n gr.LinePlot(df, x=\"time\", y=\"price\")\n gr.ScatterPlot(df, x=\"time\", y=\"price\", color=\"origin\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_temporal"
},
{
"id": 607,
"parent": 605,
"path": "06_data-science-and-plots/02_time-plots.md",
"level": 2,
"title": "Aggregating by Time",
"content": "You may wish to bin data by time buckets. Use `x_bin` to do so, using a string suffix with \"s\", \"m\", \"h\" or \"d\", such as \"15m\" or \"1d\".\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n plot = gr.BarPlot(df, x=\"time\", y=\"price\", x_bin=\"10m\")\n\n bins = gr.Radio([\"10m\", \"30m\", \"1h\"], label=\"Bin Size\")\n bins.change(lambda bins: gr.BarPlot(x_bin=bins), bins, plot)\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_aggregate_temporal"
},
{
"id": 608,
"parent": 605,
"path": "06_data-science-and-plots/02_time-plots.md",
"level": 2,
"title": "DateTime Components",
"content": "You can use `gr.DateTime` to accept input datetime data. This works well with plots for defining the x-axis range for the data.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n with gr.Row():\n start = gr.DateTime(\"now - 24h\")\n end = gr.DateTime(\"now\")\n apply_btn = gr.Button(\"Apply\")\n plot = gr.LinePlot(df, x=\"time\", y=\"price\")\n\n apply_btn.click(lambda start, end: gr.BarPlot(x_lim=[start, end]), [start, end], plot)\n \nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_datetime\n\nNote how `gr.DateTime` can accept a full datetime string, or a shorthand using `now - [0-9]+[smhd]` format to refer to a past time.\n\nYou will often have many time plots in which case you'd like to keep the x-axes in sync. The `DateTimeRange` custom component keeps a set of datetime plots in sync, and also uses the `.select` listener of plots to allow you to zoom into plots while keeping plots in sync. \n\nBecause it is a custom component, you first need to `pip install gradio_datetimerange`. Then run the following:\n\n```py\nimport gradio as gr\nfrom gradio_datetimerange import DateTimeRange\nfrom data import df\n\nwith gr.Blocks() as demo:\n daterange = DateTimeRange([\"now - 24h\", \"now\"])\n plot1 = gr.LinePlot(df, x=\"time\", y=\"price\")\n plot2 = gr.LinePlot(df, x=\"time\", y=\"price\", color=\"origin\")\n daterange.bind([plot1, plot2])\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_datetimerange\n\nTry zooming around in the plots and see how DateTimeRange updates. All the plots updates their `x_lim` in sync. You also have a \"Back\" link in the component to allow you to quickly zoom in and out."
},
{
"id": 609,
"parent": 605,
"path": "06_data-science-and-plots/02_time-plots.md",
"level": 2,
"title": "RealTime Data",
"content": "In many cases, you're working with live, realtime date, not a static dataframe. In this case, you'd update the plot regularly with a `gr.Timer()`. Assuming there's a `get_data` method that gets the latest dataframe:\n\n```python\nwith gr.Blocks() as demo:\n timer = gr.Timer(5)\n plot1 = gr.BarPlot(x=\"time\", y=\"price\")\n plot2 = gr.BarPlot(x=\"time\", y=\"price\", color=\"origin\")\n\n timer.tick(lambda: [get_data(), get_data()], outputs=[plot1, plot2])\n```\n\nYou can also use the `every` shorthand to attach a `Timer` to a component that has a function value:\n\n```python\nwith gr.Blocks() as demo:\n timer = gr.Timer(5)\n plot1 = gr.BarPlot(get_data, x=\"time\", y=\"price\", every=timer)\n plot2 = gr.BarPlot(get_data, x=\"time\", y=\"price\", color=\"origin\", every=timer)\n```"
},
{
"id": 610,
"parent": null,
"path": "06_data-science-and-plots/01_creating-plots.md",
"level": 1,
"title": "Creating Plots",
"content": "Gradio is a great way to create extremely customizable dashboards. Gradio comes with three native Plot components: `gr.LinePlot`, `gr.ScatterPlot` and `gr.BarPlot`. All these plots have the same API. Let's take a look how to set them up."
},
{
"id": 611,
"parent": 610,
"path": "06_data-science-and-plots/01_creating-plots.md",
"level": 2,
"title": "Creating a Plot with a pd.Dataframe",
"content": "Plots accept a pandas Dataframe as their value. The plot also takes `x` and `y` which represent the names of the columns that represent the x and y axes respectively. Here's a simple example:\n\n```py\nimport gradio as gr\nimport pandas as pd\nimport numpy as np\nimport random\n\ndf = pd.DataFrame({\n 'height': np.random.randint(50, 70, 25),\n 'weight': np.random.randint(120, 320, 25),\n 'age': np.random.randint(18, 65, 25),\n 'ethnicity': [random.choice([\"white\", \"black\", \"asian\"]) for _ in range(25)]\n})\n\nwith gr.Blocks() as demo:\n gr.LinePlot(df, x=\"weight\", y=\"height\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_line\n\nAll plots have the same API, so you could swap this out with a `gr.ScatterPlot`:\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n gr.ScatterPlot(df, x=\"weight\", y=\"height\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_scatter\n\nThe y axis column in the dataframe should have a numeric type, but the x axis column can be anything from strings, numbers, categories, or datetimes.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n gr.ScatterPlot(df, x=\"ethnicity\", y=\"height\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_scatter_nominal"
},
{
"id": 612,
"parent": 610,
"path": "06_data-science-and-plots/01_creating-plots.md",
"level": 2,
"title": "Breaking out Series by Color",
"content": "You can break out your plot into series using the `color` argument.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n gr.ScatterPlot(df, x=\"weight\", y=\"height\", color=\"ethnicity\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_series_nominal\n\nIf you wish to assign series specific colors, use the `color_map` arg, e.g. `gr.ScatterPlot(..., color_map={'white': '#FF9988', 'asian': '#88EEAA', 'black': '#333388'})`\n\nThe color column can be numeric type as well.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n gr.ScatterPlot(df, x=\"weight\", y=\"height\", color=\"age\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_series_quantitative"
},
{
"id": 613,
"parent": 610,
"path": "06_data-science-and-plots/01_creating-plots.md",
"level": 2,
"title": "Aggregating Values",
"content": "You can aggregate values into groups using the `x_bin` and `y_aggregate` arguments. If your x-axis is numeric, providing an `x_bin` will create a histogram-style binning:\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n gr.BarPlot(df, x=\"weight\", y=\"height\", x_bin=10, y_aggregate=\"sum\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_aggregate_quantitative\n\nIf your x-axis is a string type instead, they will act as the category bins automatically:\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n gr.BarPlot(df, x=\"ethnicity\", y=\"height\", y_aggregate=\"mean\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_aggregate_nominal"
},
{
"id": 614,
"parent": 610,
"path": "06_data-science-and-plots/01_creating-plots.md",
"level": 2,
"title": "Selecting Regions",
"content": "You can use the `.select` listener to select regions of a plot. Click and drag on the plot below to select part of the plot.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n plt = gr.LinePlot(df, x=\"weight\", y=\"height\")\n selection_total = gr.Number(label=\"Total Weight of Selection\")\n\n def select_region(selection: gr.SelectData):\n min_w, max_w = selection.index\n return df[(df[\"weight\"] >= min_w) & (df[\"weight\"] <= max_w)][\"weight\"].sum()\n\n plt.select(select_region, None, selection_total)\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_selection\n\nYou can combine this and the `.double_click` listener to create some zoom in/out effects by changing `x_lim` which sets the bounds of the x-axis:\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n plt = gr.LinePlot(df, x=\"weight\", y=\"height\")\n\n def select_region(selection: gr.SelectData):\n min_w, max_w = selection.index\n return gr.LinePlot(x_lim=(min_w, max_w)) # type: ignore\n\n plt.select(select_region, None, plt)\n plt.double_click(lambda: gr.LinePlot(x_lim=None), None, plt)\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_zoom\n\nIf you had multiple plots with the same x column, your event listeners could target the x limits of all other plots so that the x-axes stay in sync.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n plt1 = gr.LinePlot(df, x=\"weight\", y=\"height\")\n plt2 = gr.BarPlot(df, x=\"weight\", y=\"age\", x_bin=10)\n plots = [plt1, plt2]\n\n def select_region(selection: gr.SelectData):\n min_w, max_w = selection.index\n return [gr.LinePlot(x_lim=(min_w, max_w))] * len(plots) # type: ignore\n\n for plt in plots:\n plt.select(select_region, None, plots)\n plt.double_click(lambda: [gr.LinePlot(x_lim=None)] * len(plots), None, plots)\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_zoom_sync"
},
{
"id": 615,
"parent": 610,
"path": "06_data-science-and-plots/01_creating-plots.md",
"level": 2,
"title": "Making an Interactive Dashboard",
"content": "Take a look how you can have an interactive dashboard where the plots are functions of other Components.\n\n```py\nimport gradio as gr\nfrom data import df\n\nwith gr.Blocks() as demo:\n with gr.Row():\n ethnicity = gr.Dropdown([\"all\", \"white\", \"black\", \"asian\"], value=\"all\")\n max_age = gr.Slider(18, 65, value=65)\n\n def filtered_df(ethnic, age):\n _df = df if ethnic == \"all\" else df[df[\"ethnicity\"] == ethnic]\n _df = _df[_df[\"age\"] < age]\n return _df\n\n gr.ScatterPlot(filtered_df, inputs=[ethnicity, max_age], x=\"weight\", y=\"height\", title=\"Weight x Height\")\n gr.LinePlot(filtered_df, inputs=[ethnicity, max_age], x=\"age\", y=\"height\", title=\"Age x Height\")\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n$demo_plot_guide_interactive\n\nIt's that simple to filter and control the data presented in your visualization!"
},
{
"id": 616,
"parent": null,
"path": "07_streaming/02_object-detection-from-webcam-with-webrtc.md",
"level": 1,
"title": "Real Time Object Detection from a Webcam Stream with WebRTC",
"content": "Tags: VISION, STREAMING, WEBCAM\n\nIn this guide, we'll use YOLOv10 to perform real-time object detection in Gradio from a user's webcam feed. We'll utilize the latest streaming features introduced in Gradio 5.0. You can see the finished product in action below:\n\n"
},
{
"id": 617,
"parent": 616,
"path": "07_streaming/02_object-detection-from-webcam-with-webrtc.md",
"level": 2,
"title": "Setting up",
"content": "Start by installing all the dependencies. Add the following lines to a `requirements.txt` file and run `pip install -r requirements.txt`:\n\n```bash\nopencv-python\ntwilio\ngradio>=5.0\ngradio-webrtc\nonnxruntime-gpu\n```\n\nWe'll use the ONNX runtime to speed up YOLOv10 inference. This guide assumes you have access to a GPU. If you don't, change `onnxruntime-gpu` to `onnxruntime`. Without a GPU, the model will run slower, resulting in a laggy demo.\n\nWe'll use OpenCV for image manipulation and the [Gradio WebRTC](https://github.com/freddyaboulton/gradio-webrtc) custom component to use [WebRTC](https://webrtc.org/) under the hood, achieving near-zero latency.\n\n**Note**: If you want to deploy this app on any cloud provider, you'll need to use the free Twilio API for their [TURN servers](https://www.twilio.com/docs/stun-turn). Create a free account on Twilio. If you're not familiar with TURN servers, consult this [guide](https://www.twilio.com/docs/stun-turn/faq#faq-what-is-nat)."
},
{
"id": 618,
"parent": 616,
"path": "07_streaming/02_object-detection-from-webcam-with-webrtc.md",
"level": 2,
"title": "The Inference Function",
"content": "We'll download the YOLOv10 model from the Hugging Face hub and instantiate a custom inference class to use this model. \n\nThe implementation of the inference class isn't covered in this guide, but you can find the source code [here](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n/blob/main/inference.py#L9) if you're interested. This implementation borrows heavily from this [github repository](https://github.com/ibaiGorordo/ONNX-YOLOv8-Object-Detection).\n\nWe're using the `yolov10-n` variant because it has the lowest latency. See the [Performance](https://github.com/THU-MIG/yolov10?tab=readme-ov-file#performance) section of the README in the YOLOv10 GitHub repository.\n\n```python\nfrom huggingface_hub import hf_hub_download\nfrom inference import YOLOv10\n\nmodel_file = hf_hub_download(\n repo_id=\"onnx-community/yolov10n\", filename=\"onnx/model.onnx\"\n)\n\nmodel = YOLOv10(model_file)\n\ndef detection(image, conf_threshold=0.3):\n image = cv2.resize(image, (model.input_width, model.input_height))\n new_image = model.detect_objects(image, conf_threshold)\n return new_image\n```\n\nOur inference function, `detection`, accepts a numpy array from the webcam and a desired confidence threshold. Object detection models like YOLO identify many objects and assign a confidence score to each. The lower the confidence, the higher the chance of a false positive. We'll let users adjust the confidence threshold.\n\nThe function returns a numpy array corresponding to the same input image with all detected objects in bounding boxes."
},
{
"id": 619,
"parent": 616,
"path": "07_streaming/02_object-detection-from-webcam-with-webrtc.md",
"level": 2,
"title": "The Gradio Demo",
"content": "The Gradio demo is straightforward, but we'll implement a few specific features:\n\n1. Use the `WebRTC` custom component to ensure input and output are sent to/from the server with WebRTC. \n2. The [WebRTC](https://github.com/freddyaboulton/gradio-webrtc) component will serve as both an input and output component.\n3. Utilize the `time_limit` parameter of the `stream` event. This parameter sets a processing time for each user's stream. In a multi-user setting, such as on Spaces, we'll stop processing the current user's stream after this period and move on to the next. \n\nWe'll also apply custom CSS to center the webcam and slider on the page.\n\n```python\nimport gradio as gr\nfrom gradio_webrtc import WebRTC\n\ncss = \"\"\".my-group {max-width: 600px !important; max-height: 600px !important;}\n .my-column {display: flex !important; justify-content: center !important; align-items: center !important;}\"\"\"\n\nwith gr.Blocks(css=css) as demo:\n gr.HTML(\n \"\"\"\n \n YOLOv10 Webcam Stream (Powered by WebRTC ⚡️)\n
\n \"\"\"\n )\n with gr.Column(elem_classes=[\"my-column\"]):\n with gr.Group(elem_classes=[\"my-group\"]):\n image = WebRTC(label=\"Stream\", rtc_configuration=rtc_configuration)\n conf_threshold = gr.Slider(\n label=\"Confidence Threshold\",\n minimum=0.0,\n maximum=1.0,\n step=0.05,\n value=0.30,\n )\n\n image.stream(\n fn=detection, inputs=[image, conf_threshold], outputs=[image], time_limit=10\n )\n\nif __name__ == \"__main__\":\n demo.launch()\n```" }, { "id": 620, "parent": 616, "path": "07_streaming/02_object-detection-from-webcam-with-webrtc.md", "level": 2, "title": "Conclusion", "content": "Our app is hosted on Hugging Face Spaces [here](https://huggingface.co/spaces/freddyaboulton/webrtc-yolov10n). \n\nYou can use this app as a starting point to build real-time image applications with Gradio. Don't hesitate to open issues in the space or in the [WebRTC component GitHub repo](https://github.com/freddyaboulton/gradio-webrtc) if you have any questions or encounter problems." }, { "id": 621, "parent": null, "path": "07_streaming/03_object-detection-from-video.md", "level": 1, "title": "Streaming Object Detection from Video", "content": "Tags: VISION, STREAMING, VIDEO\n\nIn this guide we'll use the [RT-DETR](https://huggingface.co/docs/transformers/en/model_doc/rt_detr) model to detect objects in a user uploaded video. We'll stream the results from the server using the new video streaming features introduced in Gradio 5.0.\n\n" }, { "id": 622, "parent": 621, "path": "07_streaming/03_object-detection-from-video.md", "level": 2, "title": "Setting up the Model", "content": "First, we'll install the following requirements in our system:\n\n```\nopencv-python\ntorch\ntransformers>=4.43.0\nspaces\n```\n\nThen, we'll download the model from the Hugging Face Hub:\n\n```python\nfrom transformers import RTDetrForObjectDetection, RTDetrImageProcessor\n\nimage_processor = RTDetrImageProcessor.from_pretrained(\"PekingU/rtdetr_r50vd\")\nmodel = RTDetrForObjectDetection.from_pretrained(\"PekingU/rtdetr_r50vd\").to(\"cuda\")\n```\n\nWe're moving the model to the GPU. We'll be deploying our model to Hugging Face Spaces and running the inference in the [free ZeroGPU cluster](https://huggingface.co/zero-gpu-explorers)." }, { "id": 623, "parent": 621, "path": "07_streaming/03_object-detection-from-video.md", "level": 2, "title": "The Inference Function", "content": "Our inference function will accept a video and a desired confidence threshold.\nObject detection models identify many objects and assign a confidence score to each object. The lower the confidence, the higher the chance of a false positive. So we will let our users set the conference threshold.\n\nOur function will iterate over the frames in the video and run the RT-DETR model over each frame.\nWe will then draw the bounding boxes for each detected object in the frame and save the frame to a new output video.\nThe function will yield each output video in chunks of two seconds.\n\nIn order to keep inference times as low as possible on ZeroGPU (there is a time-based quota),\nwe will halve the original frames-per-second in the output video and resize the input frames to be half the original \nsize before running the model.\n\nThe code for the inference function is below - we'll go over it piece by piece.\n\n```python\nimport spaces\nimport cv2\nfrom PIL import Image\nimport torch\nimport time\nimport numpy as np\nimport uuid\n\nfrom draw_boxes import draw_bounding_boxes\n\nSUBSAMPLE = 2\n\n@spaces.GPU\ndef stream_object_detection(video, conf_threshold):\n cap = cv2.VideoCapture(video)\n\n # This means we will output mp4 videos\n video_codec = cv2.VideoWriter_fourcc(*\"mp4v\") # type: ignore\n fps = int(cap.get(cv2.CAP_PROP_FPS))\n\n desired_fps = fps // SUBSAMPLE\n width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) // 2\n height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) // 2\n\n iterating, frame = cap.read()\n\n n_frames = 0\n\n # Use UUID to create a unique video file\n output_video_name = f\"output_{uuid.uuid4()}.mp4\"\n\n # Output Video\n output_video = cv2.VideoWriter(output_video_name, video_codec, desired_fps, (width, height)) # type: ignore\n batch = []\n\n while iterating:\n frame = cv2.resize( frame, (0,0), fx=0.5, fy=0.5)\n frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n if n_frames % SUBSAMPLE == 0:\n batch.append(frame)\n if len(batch) == 2 * desired_fps:\n inputs = image_processor(images=batch, return_tensors=\"pt\").to(\"cuda\")\n\n with torch.no_grad():\n outputs = model(**inputs)\n\n boxes = image_processor.post_process_object_detection(\n outputs,\n target_sizes=torch.tensor([(height, width)] * len(batch)),\n threshold=conf_threshold)\n \n for i, (array, box) in enumerate(zip(batch, boxes)):\n pil_image = draw_bounding_boxes(Image.fromarray(array), box, model, conf_threshold)\n frame = np.array(pil_image)\n # Convert RGB to BGR\n frame = frame[:, :, ::-1].copy()\n output_video.write(frame)\n\n batch = []\n output_video.release()\n yield output_video_name\n output_video_name = f\"output_{uuid.uuid4()}.mp4\"\n output_video = cv2.VideoWriter(output_video_name, video_codec, desired_fps, (width, height)) # type: ignore\n\n iterating, frame = cap.read()\n n_frames += 1\n```\n\n1. **Reading from the Video**\n\nOne of the industry standards for creating videos in python is OpenCV so we will use it in this app.\n\nThe `cap` variable is how we will read from the input video. Whenever we call `cap.read()`, we are reading the next frame in the video.\n\nIn order to stream video in Gradio, we need to yield a different video file for each \"chunk\" of the output video.\nWe create the next video file to write to with the `output_video = cv2.VideoWriter(output_video_name, video_codec, desired_fps, (width, height))` line. The `video_codec` is how we specify the type of video file. Only \"mp4\" and \"ts\" files are supported for video sreaming at the moment.\n\n\n2. **The Inference Loop**\n\nFor each frame in the video, we will resize it to be half the size. OpenCV reads files in `BGR` format, so will convert to the expected `RGB` format of transfomers. That's what the first two lines of the while loop are doing. \n\nWe take every other frame and add it to a `batch` list so that the output video is half the original FPS. When the batch covers two seconds of video, we will run the model. The two second threshold was chosen to keep the processing time of each batch small enough so that video is smoothly displayed in the server while not requiring too many separate forward passes. In order for video streaming to work properly in Gradio, the batch size should be at least 1 second. \n\nWe run the forward pass of the model and then use the `post_process_object_detection` method of the model to scale the detected bounding boxes to the size of the input frame.\n\nWe make use of a custom function to draw the bounding boxes (source [here](https://huggingface.co/spaces/gradio/rt-detr-object-detection/blob/main/draw_boxes.py#L14)). We then have to convert from `RGB` to `BGR` before writing back to the output video.\n\nOnce we have finished processing the batch, we create a new output video file for the next batch." }, { "id": 624, "parent": 621, "path": "07_streaming/03_object-detection-from-video.md", "level": 2, "title": "The Gradio Demo", "content": "The UI code is pretty similar to other kinds of Gradio apps. \nWe'll use a standard two-column layout so that users can see the input and output videos side by side.\n\nIn order for streaming to work, we have to set `streaming=True` in the output video. Setting the video\nto autoplay is not necessary but it's a better experience for users.\n\n```python\nimport gradio as gr\n\nwith gr.Blocks() as app:\n gr.HTML(\n \"\"\"\n\n Video Object Detection with RT-DETR\n
\n \"\"\")\n with gr.Row():\n with gr.Column():\n video = gr.Video(label=\"Video Source\")\n conf_threshold = gr.Slider(\n label=\"Confidence Threshold\",\n minimum=0.0,\n maximum=1.0,\n step=0.05,\n value=0.30,\n )\n with gr.Column():\n output_video = gr.Video(label=\"Processed Video\", streaming=True, autoplay=True)\n\n video.upload(\n fn=stream_object_detection,\n inputs=[video, conf_threshold],\n outputs=[output_video],\n )\n\n\n```" }, { "id": 625, "parent": 621, "path": "07_streaming/03_object-detection-from-video.md", "level": 2, "title": "Conclusion", "content": "You can check out our demo hosted on Hugging Face Spaces [here](https://huggingface.co/spaces/gradio/rt-detr-object-detection). \n\nIt is also embedded on this page below\n\n$demo_rt-detr-object-detection" }, { "id": 626, "parent": null, "path": "07_streaming/04_conversational-chatbot.md", "level": 1, "title": "Building Conversational Chatbots with Gradio", "content": "Tags: AUDIO, STREAMING, CHATBOTS" }, { "id": 627, "parent": 626, "path": "07_streaming/04_conversational-chatbot.md", "level": 2, "title": "Introduction", "content": "The next generation of AI user interfaces is moving towards audio-native experiences. Users will be able to speak to chatbots and receive spoken responses in return. Several models have been built under this paradigm, including GPT-4o and [mini omni](https://github.com/gpt-omni/mini-omni).\n\nIn this guide, we'll walk you through building your own conversational chat application using mini omni as an example. You can see a demo of the finished app below:\n\n" }, { "id": 628, "parent": 626, "path": "07_streaming/04_conversational-chatbot.md", "level": 2, "title": "Application Overview", "content": "Our application will enable the following user experience:\n\n1. Users click a button to start recording their message\n2. The app detects when the user has finished speaking and stops recording\n3. The user's audio is passed to the omni model, which streams back a response\n4. After omni mini finishes speaking, the user's microphone is reactivated\n5. All previous spoken audio, from both the user and omni, is displayed in a chatbot component\n\nLet's dive into the implementation details." }, { "id": 629, "parent": 626, "path": "07_streaming/04_conversational-chatbot.md", "level": 2, "title": "Processing User Audio", "content": "We'll stream the user's audio from their microphone to the server and determine if the user has stopped speaking on each new chunk of audio.\n\nHere's our `process_audio` function:\n\n```python\nimport numpy as np\nfrom utils import determine_pause\n\ndef process_audio(audio: tuple, state: AppState):\n if state.stream is None:\n state.stream = audio[1]\n state.sampling_rate = audio[0]\n else:\n state.stream = np.concatenate((state.stream, audio[1]))\n\n pause_detected = determine_pause(state.stream, state.sampling_rate, state)\n state.pause_detected = pause_detected\n\n if state.pause_detected and state.started_talking:\n return gr.Audio(recording=False), state\n return None, state\n```\n\nThis function takes two inputs:\n1. The current audio chunk (a tuple of `(sampling_rate, numpy array of audio)`)\n2. The current application state\n\nWe'll use the following `AppState` dataclass to manage our application state:\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass AppState:\n stream: np.ndarray | None = None\n sampling_rate: int = 0\n pause_detected: bool = False\n stopped: bool = False\n conversation: list = []\n```\n\nThe function concatenates new audio chunks to the existing stream and checks if the user has stopped speaking. If a pause is detected, it returns an update to stop recording. Otherwise, it returns `None` to indicate no changes.\n\nThe implementation of the `determine_pause` function is specific to the omni-mini project and can be found [here](https://huggingface.co/spaces/gradio/omni-mini/blob/eb027808c7bfe5179b46d9352e3fa1813a45f7c3/app.py#L98)." }, { "id": 630, "parent": 626, "path": "07_streaming/04_conversational-chatbot.md", "level": 2, "title": "Generating the Response", "content": "After processing the user's audio, we need to generate and stream the chatbot's response. Here's our `response` function:\n\n```python\nimport io\nimport tempfile\nfrom pydub import AudioSegment\n\ndef response(state: AppState):\n if not state.pause_detected and not state.started_talking:\n return None, AppState()\n \n audio_buffer = io.BytesIO()\n\n segment = AudioSegment(\n state.stream.tobytes(),\n frame_rate=state.sampling_rate,\n sample_width=state.stream.dtype.itemsize,\n channels=(1 if len(state.stream.shape) == 1 else state.stream.shape[1]),\n )\n segment.export(audio_buffer, format=\"wav\")\n\n with tempfile.NamedTemporaryFile(suffix=\".wav\", delete=False) as f:\n f.write(audio_buffer.getvalue())\n \n state.conversation.append({\"role\": \"user\",\n \"content\": {\"path\": f.name,\n \"mime_type\": \"audio/wav\"}})\n \n output_buffer = b\"\"\n\n for mp3_bytes in speaking(audio_buffer.getvalue()):\n output_buffer += mp3_bytes\n yield mp3_bytes, state\n\n with tempfile.NamedTemporaryFile(suffix=\".mp3\", delete=False) as f:\n f.write(output_buffer)\n \n state.conversation.append({\"role\": \"assistant\",\n \"content\": {\"path\": f.name,\n \"mime_type\": \"audio/mp3\"}})\n yield None, AppState(conversation=state.conversation)\n```\n\nThis function:\n1. Converts the user's audio to a WAV file\n2. Adds the user's message to the conversation history\n3. Generates and streams the chatbot's response using the `speaking` function\n4. Saves the chatbot's response as an MP3 file\n5. Adds the chatbot's response to the conversation history\n\nNote: The implementation of the `speaking` function is specific to the omni-mini project and can be found [here](https://huggingface.co/spaces/gradio/omni-mini/blob/main/app.py#L116)." }, { "id": 631, "parent": 626, "path": "07_streaming/04_conversational-chatbot.md", "level": 2, "title": "Building the Gradio App", "content": "Now let's put it all together using Gradio's Blocks API:\n\n```python\nimport gradio as gr\n\ndef start_recording_user(state: AppState):\n if not state.stopped:\n return gr.Audio(recording=True)\n\nwith gr.Blocks() as demo:\n with gr.Row():\n with gr.Column():\n input_audio = gr.Audio(\n label=\"Input Audio\", sources=\"microphone\", type=\"numpy\"\n )\n with gr.Column():\n chatbot = gr.Chatbot(label=\"Conversation\", type=\"messages\")\n output_audio = gr.Audio(label=\"Output Audio\", streaming=True, autoplay=True)\n state = gr.State(value=AppState())\n\n stream = input_audio.stream(\n process_audio,\n [input_audio, state],\n [input_audio, state],\n stream_every=0.5,\n time_limit=30,\n )\n respond = input_audio.stop_recording(\n response,\n [state],\n [output_audio, state]\n )\n respond.then(lambda s: s.conversation, [state], [chatbot])\n\n restart = output_audio.stop(\n start_recording_user,\n [state],\n [input_audio]\n )\n cancel = gr.Button(\"Stop Conversation\", variant=\"stop\")\n cancel.click(lambda: (AppState(stopped=True), gr.Audio(recording=False)), None,\n [state, input_audio], cancels=[respond, restart])\n\nif __name__ == \"__main__\":\n demo.launch()\n```\n\nThis setup creates a user interface with:\n- An input audio component for recording user messages\n- A chatbot component to display the conversation history\n- An output audio component for the chatbot's responses\n- A button to stop and reset the conversation\n\nThe app streams user audio in 0.5-second chunks, processes it, generates responses, and updates the conversation history accordingly." }, { "id": 632, "parent": 626, "path": "07_streaming/04_conversational-chatbot.md", "level": 2, "title": "Conclusion", "content": "This guide demonstrates how to build a conversational chatbot application using Gradio and the mini omni model. You can adapt this framework to create various audio-based chatbot demos. To see the full application in action, visit the Hugging Face Spaces demo: https://huggingface.co/spaces/gradio/omni-mini\n\nFeel free to experiment with different models, audio processing techniques, or user interface designs to create your own unique conversational AI experiences!" }, { "id": 633, "parent": null, "path": "07_streaming/05_real-time-speech-recognition.md", "level": 1, "title": "Real Time Speech Recognition", "content": "Tags: ASR, SPEECH, STREAMING" }, { "id": 634, "parent": 633, "path": "07_streaming/05_real-time-speech-recognition.md", "level": 2, "title": "Introduction", "content": "Automatic speech recognition (ASR), the conversion of spoken speech to text, is a very important and thriving area of machine learning. ASR algorithms run on practically every smartphone, and are becoming increasingly embedded in professional workflows, such as digital assistants for nurses and doctors. Because ASR algorithms are designed to be used directly by customers and end users, it is important to validate that they are behaving as expected when confronted with a wide variety of speech patterns (different accents, pitches, and background audio conditions).\n\nUsing `gradio`, you can easily build a demo of your ASR model and share that with a testing team, or test it yourself by speaking through the microphone on your device.\n\nThis tutorial will show how to take a pretrained speech-to-text model and deploy it with a Gradio interface. We will start with a **_full-context_** model, in which the user speaks the entire audio before the prediction runs. Then we will adapt the demo to make it **_streaming_**, meaning that the audio model will convert speech as you speak." }, { "id": 635, "parent": 634, "path": "07_streaming/05_real-time-speech-recognition.md", "level": 3, "title": "Prerequisites", "content": "Make sure you have the `gradio` Python package already [installed](/getting_started). You will also need a pretrained speech recognition model. In this tutorial, we will build demos from 2 ASR libraries:\n\n- Transformers (for this, `pip install torch transformers torchaudio`)\n\nMake sure you have at least one of these installed so that you can follow along the tutorial. You will also need `ffmpeg` [installed on your system](https://www.ffmpeg.org/download.html), if you do not already have it, to process files from the microphone.\n\nHere's how to build a real time speech recognition (ASR) app:\n\n1. [Set up the Transformers ASR Model](#1-set-up-the-transformers-asr-model)\n2. [Create a Full-Context ASR Demo with Transformers](#2-create-a-full-context-asr-demo-with-transformers)\n3. [Create a Streaming ASR Demo with Transformers](#3-create-a-streaming-asr-demo-with-transformers)" }, { "id": 636, "parent": 633, "path": "07_streaming/05_real-time-speech-recognition.md", "level": 2, "title": "1. Set up the Transformers ASR Model", "content": "First, you will need to have an ASR model that you have either trained yourself or you will need to download a pretrained model. In this tutorial, we will start by using a pretrained ASR model from the model, `whisper`.\n\nHere is the code to load `whisper` from Hugging Face `transformers`.\n\n```python\nfrom transformers import pipeline\n\np = pipeline(\"automatic-speech-recognition\", model=\"openai/whisper-base.en\")\n```\n\nThat's it!" }, { "id": 637, "parent": 633, "path": "07_streaming/05_real-time-speech-recognition.md", "level": 2, "title": "2. Create a Full-Context ASR Demo with Transformers", "content": "We will start by creating a _full-context_ ASR demo, in which the user speaks the full audio before using the ASR model to run inference. This is very easy with Gradio -- we simply create a function around the `pipeline` object above.\n\nWe will use `gradio`'s built in `Audio` component, configured to take input from the user's microphone and return a filepath for the recorded audio. The output component will be a plain `Textbox`.\n\n```py\nimport gradio as gr\nfrom transformers import pipeline\nimport numpy as np\n\ntranscriber = pipeline(\"automatic-speech-recognition\", model=\"openai/whisper-base.en\")\n\ndef transcribe(audio):\n sr, y = audio\n \n # Convert to mono if stereo\n if y.ndim > 1:\n y = y.mean(axis=1)\n \n y = y.astype(np.float32)\n y /= np.max(np.abs(y))\n\n return transcriber({\"sampling_rate\": sr, \"raw\": y})[\"text\"] # type: ignore\n\ndemo = gr.Interface(\n transcribe,\n gr.Audio(sources=\"microphone\"),\n \"text\",\n)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n$demo_asr\n\nThe `transcribe` function takes a single parameter, `audio`, which is a numpy array of the audio the user recorded. The `pipeline` object expects this in float32 format, so we convert it first to float32, and then extract the transcribed text." }, { "id": 638, "parent": 633, "path": "07_streaming/05_real-time-speech-recognition.md", "level": 2, "title": "3. Create a Streaming ASR Demo with Transformers", "content": "To make this a *streaming* demo, we need to make these changes:\n\n1. Set `streaming=True` in the `Audio` component\n2. Set `live=True` in the `Interface`\n3. Add a `state` to the interface to store the recorded audio of a user\n\nTip: You can also set `time_limit` and `stream_every` parameters in the interface. The `time_limit` caps the amount of time each user's stream can take. The default is 30 seconds so users won't be able to stream audio for more than 30 seconds. The `stream_every` parameter controls how frequently data is sent to your function. By default it is 0.5 seconds.\n\nTake a look below.\n\n```py\nimport gradio as gr\nfrom transformers import pipeline\nimport numpy as np\n\ntranscriber = pipeline(\"automatic-speech-recognition\", model=\"openai/whisper-base.en\")\n\ndef transcribe(stream, new_chunk):\n sr, y = new_chunk\n \n # Convert to mono if stereo\n if y.ndim > 1:\n y = y.mean(axis=1)\n \n y = y.astype(np.float32)\n y /= np.max(np.abs(y))\n\n if stream is not None:\n stream = np.concatenate([stream, y])\n else:\n stream = y\n return stream, transcriber({\"sampling_rate\": sr, \"raw\": stream})[\"text\"] # type: ignore\n\ndemo = gr.Interface(\n transcribe,\n [\"state\", gr.Audio(sources=[\"microphone\"], streaming=True)],\n [\"state\", \"text\"],\n live=True,\n)\n\nif __name__ == \"__main__\":\n demo.launch()\n\n```\n\nNotice that we now have a state variable because we need to track all the audio history. `transcribe` gets called whenever there is a new small chunk of audio, but we also need to keep track of all the audio spoken so far in the state. As the interface runs, the `transcribe` function gets called, with a record of all the previously spoken audio in the `stream` and the new chunk of audio as `new_chunk`. We return the new full audio to be stored back in its current state, and we also return the transcription. Here, we naively append the audio together and call the `transcriber` object on the entire audio. You can imagine more efficient ways of handling this, such as re-processing only the last 5 seconds of audio whenever a new chunk of audio is received. \n\n$demo_stream_asr\n\nNow the ASR model will run inference as you speak!" }, { "id": 639, "parent": null, "path": "07_streaming/01_streaming-ai-generated-audio.md", "level": 1, "title": "Streaming AI Generated Audio", "content": "Tags: AUDIO, STREAMING\n\nIn this guide, we'll build a novel AI application to showcase Gradio's audio output streaming. We're going to a build a talking [Magic 8 Ball](https://en.wikipedia.org/wiki/Magic_8_Ball) 🎱\n\nA Magic 8 Ball is a toy that answers any question after you shake it. Our application will do the same but it will also speak its response!\n\nWe won't cover all the implementation details in this blog post but the code is freely available on [Hugging Face Spaces](https://huggingface.co/spaces/gradio/magic-8-ball)." }, { "id": 640, "parent": 639, "path": "07_streaming/01_streaming-ai-generated-audio.md", "level": 2, "title": "The Overview", "content": "Just like the classic Magic 8 Ball, a user should ask it a question orally and then wait for a response. Under the hood, we'll use Whisper to transcribe the audio and then use an LLM to generate a magic-8-ball-style answer. Finally, we'll use Parler TTS to read the response aloud." }, { "id": 641, "parent": 639, "path": "07_streaming/01_streaming-ai-generated-audio.md", "level": 2, "title": "The UI", "content": "First let's define the UI and put placeholders for all the python logic.\n\n```python\nimport gradio as gr\n\nwith gr.Blocks() as block:\n gr.HTML(\n f\"\"\"\nMagic 8 Ball 🎱
\nAsk a question and receive wisdom
\nPowered by Parler-TTS\n \"\"\"\n )\n with gr.Group():\n with gr.Row():\n audio_out = gr.Audio(label=\"Spoken Answer\", streaming=True, autoplay=True)\n answer = gr.Textbox(label=\"Answer\")\n state = gr.State()\n with gr.Row():\n audio_in = gr.Audio(label=\"Speak your question\", sources=\"microphone\", type=\"filepath\")\n\n audio_in.stop_recording(generate_response, audio_in, [state, answer, audio_out])\\\n .then(fn=read_response, inputs=state, outputs=[answer, audio_out])\n\nblock.launch()\n```\n\nWe're placing the output Audio and Textbox components and the input Audio component in separate rows. In order to stream the audio from the server, we'll set `streaming=True` in the output Audio component. We'll also set `autoplay=True` so that the audio plays as soon as it's ready.\nWe'll be using the Audio input component's `stop_recording` event to trigger our application's logic when a user stops recording from their microphone.\n\nWe're separating the logic into two parts. First, `generate_response` will take the recorded audio, transcribe it and generate a response with an LLM. We're going to store the response in a `gr.State` variable that then gets passed to the `read_response` function that generates the audio.\n\nWe're doing this in two parts because only `read_response` will require a GPU. Our app will run on Hugging Faces [ZeroGPU](https://huggingface.co/zero-gpu-explorers) which has time-based quotas. Since generating the response can be done with Hugging Face's Inference API, we shouldn't include that code in our GPU function as it will needlessly use our GPU quota." }, { "id": 642, "parent": 639, "path": "07_streaming/01_streaming-ai-generated-audio.md", "level": 2, "title": "The Logic", "content": "As mentioned above, we'll use [Hugging Face's Inference API](https://huggingface.co/docs/huggingface_hub/guides/inference) to transcribe the audio and generate a response from an LLM. After instantiating the client, I use the `automatic_speech_recognition` method (this automatically uses Whisper running on Hugging Face's Inference Servers) to transcribe the audio. Then I pass the question to an LLM (Mistal-7B-Instruct) to generate a response. We are prompting the LLM to act like a magic 8 ball with the system message.\n\nOur `generate_response` function will also send empty updates to the output textbox and audio components (returning `None`). \nThis is because I want the Gradio progress tracker to be displayed over the components but I don't want to display the answer until the audio is ready.\n\n\n```python\nfrom huggingface_hub import InferenceClient\n\nclient = InferenceClient(token=os.getenv(\"HF_TOKEN\"))\n\ndef generate_response(audio):\n gr.Info(\"Transcribing Audio\", duration=5)\n question = client.automatic_speech_recognition(audio).text\n\n messages = [{\"role\": \"system\", \"content\": (\"You are a magic 8 ball.\"\n \"Someone will present to you a situation or question and your job \"\n \"is to answer with a cryptic adage or proverb such as \"\n \"'curiosity killed the cat' or 'The early bird gets the worm'.\"\n \"Keep your answers short and do not include the phrase 'Magic 8 Ball' in your response. If the question does not make sense or is off-topic, say 'Foolish questions get foolish answers.'\"\n \"For example, 'Magic 8 Ball, should I get a dog?', 'A dog is ready for you but are you ready for the dog?'\")},\n {\"role\": \"user\", \"content\": f\"Magic 8 Ball please answer this question - {question}\"}]\n \n response = client.chat_completion(messages, max_tokens=64, seed=random.randint(1, 5000),\n model=\"mistralai/Mistral-7B-Instruct-v0.3\")\n\n response = response.choices[0].message.content.replace(\"Magic 8 Ball\", \"\").replace(\":\", \"\")\n return response, None, None\n```\n\n\nNow that we have our text response, we'll read it aloud with Parler TTS. The `read_response` function will be a python generator that yields the next chunk of audio as it's ready.\n\n\nWe'll be using the [Mini v0.1](https://huggingface.co/parler-tts/parler_tts_mini_v0.1) for the feature extraction but the [Jenny fine tuned version](https://huggingface.co/parler-tts/parler-tts-mini-jenny-30H) for the voice. This is so that the voice is consistent across generations.\n\n\nStreaming audio with transformers requires a custom Streamer class. You can see the implementation [here](https://huggingface.co/spaces/gradio/magic-8-ball/blob/main/streamer.py). Additionally, we'll convert the output to bytes so that it can be streamed faster from the backend. \n\n\n```python\nfrom streamer import ParlerTTSStreamer\nfrom transformers import AutoTokenizer, AutoFeatureExtractor, set_seed\nimport numpy as np\nimport spaces\nimport torch\nfrom threading import Thread\n\n\ndevice = \"cuda:0\" if torch.cuda.is_available() else \"mps\" if torch.backends.mps.is_available() else \"cpu\"\ntorch_dtype = torch.float16 if device != \"cpu\" else torch.float32\n\nrepo_id = \"parler-tts/parler_tts_mini_v0.1\"\n\njenny_repo_id = \"ylacombe/parler-tts-mini-jenny-30H\"\n\nmodel = ParlerTTSForConditionalGeneration.from_pretrained(\n jenny_repo_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True\n).to(device)\n\ntokenizer = AutoTokenizer.from_pretrained(repo_id)\nfeature_extractor = AutoFeatureExtractor.from_pretrained(repo_id)\n\nsampling_rate = model.audio_encoder.config.sampling_rate\nframe_rate = model.audio_encoder.config.frame_rate\n\n@spaces.GPU\ndef read_response(answer):\n\n play_steps_in_s = 2.0\n play_steps = int(frame_rate * play_steps_in_s)\n\n description = \"Jenny speaks at an average pace with a calm delivery in a very confined sounding environment with clear audio quality.\"\n description_tokens = tokenizer(description, return_tensors=\"pt\").to(device)\n\n streamer = ParlerTTSStreamer(model, device=device, play_steps=play_steps)\n prompt = tokenizer(answer, return_tensors=\"pt\").to(device)\n\n generation_kwargs = dict(\n input_ids=description_tokens.input_ids,\n prompt_input_ids=prompt.input_ids,\n streamer=streamer,\n do_sample=True,\n temperature=1.0,\n min_new_tokens=10,\n )\n\n set_seed(42)\n thread = Thread(target=model.generate, kwargs=generation_kwargs)\n thread.start()\n\n for new_audio in streamer:\n print(f\"Sample of length: {round(new_audio.shape[0] / sampling_rate, 2)} seconds\")\n yield answer, numpy_to_mp3(new_audio, sampling_rate=sampling_rate)\n```" }, { "id": 643, "parent": 639, "path": "07_streaming/01_streaming-ai-generated-audio.md", "level": 2, "title": "Conclusion", "content": "You can see our final application [here](https://huggingface.co/spaces/gradio/magic-8-ball)!" }, { "id": 644, "parent": null, "path": "07_streaming/06_automatic-voice-detection.md", "level": 1, "title": "Multimodal Gradio App Powered by Groq with Automatic Speech Detection", "content": "Tags: AUDIO, STREAMING, CHATBOTS, VOICE" }, { "id": 645, "parent": 644, "path": "07_streaming/06_automatic-voice-detection.md", "level": 2, "title": "Introduction", "content": "Modern voice applications should feel natural and responsive, moving beyond the traditional \"click-to-record\" pattern. By combining Groq's fast inference capabilities with automatic speech detection, we can create a more intuitive interaction model where users can simply start talking whenever they want to engage with the AI.\n\n> Credits: VAD and Gradio code inspired by [WillHeld's Diva-audio-chat](https://huggingface.co/spaces/WillHeld/diva-audio-chat/tree/main).\n\nIn this tutorial, you will learn how to create a multimodal Gradio and Groq app that has automatic speech detection. You can also watch the full video tutorial which includes a demo of the application:\n\n" }, { "id": 646, "parent": 644, "path": "07_streaming/06_automatic-voice-detection.md", "level": 2, "title": "Background", "content": "Many voice apps currently work by the user clicking record, speaking, then stopping the recording. While this can be a powerful demo, the most natural mode of interaction with voice requires the app to dynamically detect when the user is speaking, so they can talk back and forth without having to continually click a record button. \n\nCreating a natural interaction with voice and text requires a dynamic and low-latency response. Thus, we need both automatic voice detection and fast inference. With @ricky0123/vad-web powering speech detection and Groq powering the LLM, both of these requirements are met. Groq provides a lightning fast response, and Gradio allows for easy creation of impressively functional apps.\n\nThis tutorial shows you how to build a calorie tracking app where you speak to an AI that automatically detects when you start and stop your response, and provides its own text response back to guide you with questions that allow it to give a calorie estimate of your last meal." }, { "id": 647, "parent": 644, "path": "07_streaming/06_automatic-voice-detection.md", "level": 2, "title": "Key Components", "content": "- **Gradio**: Provides the web interface and audio handling capabilities\n- **@ricky0123/vad-web**: Handles voice activity detection\n- **Groq**: Powers fast LLM inference for natural conversations\n- **Whisper**: Transcribes speech to text" }, { "id": 648, "parent": 647, "path": "07_streaming/06_automatic-voice-detection.md", "level": 3, "title": "Setting Up the Environment", "content": "First, let’s install and import our essential libraries and set up a client for using the Groq API. Here’s how to do it:\n\n`requirements.txt`\n```\ngradio\ngroq\nnumpy\nsoundfile\nlibrosa\nspaces\nxxhash\ndatasets\n```\n\n`app.py`\n```python\nimport groq\nimport gradio as gr\nimport soundfile as sf\nfrom dataclasses import dataclass, field\nimport os" }, { "id": 649, "parent": null, "path": "07_streaming/06_automatic-voice-detection.md", "level": 1, "title": "Initialize Groq client securely", "content": "api_key = os.environ.get(\"GROQ_API_KEY\")\nif not api_key:\n raise ValueError(\"Please set the GROQ_API_KEY environment variable.\")\nclient = groq.Client(api_key=api_key)\n```\n\nHere, we’re pulling in key libraries to interact with the Groq API, build a sleek UI with Gradio, and handle audio data. We’re accessing the Groq API key securely with a key stored in an environment variable, which is a security best practice for avoiding leaking the API key.\n\n---" }, { "id": 650, "parent": 649, "path": "07_streaming/06_automatic-voice-detection.md", "level": 3, "title": "State Management for Seamless Conversations", "content": "We need a way to keep track of our conversation history, so the chatbot remembers past interactions, and manage other states like whether recording is currently active. To do this, let’s create an `AppState` class:\n\n```python\n@dataclass\nclass AppState:\n conversation: list = field(default_factory=list)\n stopped: bool = False\n model_outs: Any = None\n```\n\nOur `AppState` class is a handy tool for managing conversation history and tracking whether recording is on or off. Each instance will have its own fresh list of conversations, making sure chat history is isolated to each session. \n\n---" }, { "id": 651, "parent": 649, "path": "07_streaming/06_automatic-voice-detection.md", "level": 3, "title": "Transcribing Audio with Whisper on Groq", "content": "Next, we’ll create a function to transcribe the user’s audio input into text using Whisper, a powerful transcription model hosted on Groq. This transcription will also help us determine whether there’s meaningful speech in the input. Here’s how:\n\n```python\ndef transcribe_audio(client, file_name):\n if file_name is None:\n return None\n\n try:\n with open(file_name, \"rb\") as audio_file:\n response = client.audio.transcriptions.with_raw_response.create(\n model=\"whisper-large-v3-turbo\",\n file=(\"audio.wav\", audio_file),\n response_format=\"verbose_json\",\n )\n completion = process_whisper_response(response.parse())\n return completion\n except Exception as e:\n print(f\"Error in transcription: {e}\")\n return f\"Error in transcription: {str(e)}\"\n```\n\nThis function opens the audio file and sends it to Groq’s Whisper model for transcription, requesting detailed JSON output. verbose_json is needed to get information to determine if speech was included in the audio. We also handle any potential errors so our app doesn’t fully crash if there’s an issue with the API request. \n\n```python\ndef process_whisper_response(completion):\n \"\"\"\n Process Whisper transcription response and return text or null based on no_speech_prob\n \n Args:\n completion: Whisper transcription response object\n \n Returns:\n str or None: Transcribed text if no_speech_prob <= 0.7, otherwise None\n \"\"\"\n if completion.segments and len(completion.segments) > 0:\n no_speech_prob = completion.segments[0].get('no_speech_prob', 0)\n print(\"No speech prob:\", no_speech_prob)\n\n if no_speech_prob > 0.7:\n return None\n \n return completion.text.strip()\n \n return None\n```\n\nWe also need to interpret the audio data response. The process_whisper_response function takes the resulting completion from Whisper and checks if the audio was just background noise or had actual speaking that was transcribed. It uses a threshold of 0.7 to interpret the no_speech_prob, and will return None if there was no speech. Otherwise, it will return the text transcript of the conversational response from the human.\n\n\n---" }, { "id": 652, "parent": 649, "path": "07_streaming/06_automatic-voice-detection.md", "level": 3, "title": "Adding Conversational Intelligence with LLM Integration", "content": "Our chatbot needs to provide intelligent, friendly responses that flow naturally. We’ll use a Groq-hosted Llama-3.2 for this:\n\n```python\ndef generate_chat_completion(client, history):\n messages = []\n messages.append(\n {\n \"role\": \"system\",\n \"content\": \"In conversation with the user, ask questions to estimate and provide (1) total calories, (2) protein, carbs, and fat in grams, (3) fiber and sugar content. Only ask *one question at a time*. Be conversational and natural.\",\n }\n )\n\n for message in history:\n messages.append(message)\n\n try:\n completion = client.chat.completions.create(\n model=\"llama-3.2-11b-vision-preview\",\n messages=messages,\n )\n return completion.choices[0].message.content\n except Exception as e:\n return f\"Error in generating chat completion: {str(e)}\"\n```\n\nWe’re defining a system prompt to guide the chatbot’s behavior, ensuring it asks one question at a time and keeps things conversational. This setup also includes error handling to ensure the app gracefully manages any issues.\n\n---" }, { "id": 653, "parent": 649, "path": "07_streaming/06_automatic-voice-detection.md", "level": 3, "title": "Voice Activity Detection for Hands-Free Interaction", "content": "To make our chatbot hands-free, we’ll add Voice Activity Detection (VAD) to automatically detect when someone starts or stops speaking. Here’s how to implement it using ONNX in JavaScript:\n\n```javascript\nasync function main() {\n const script1 = document.createElement(\"script\");\n script1.src = \"https://cdn.jsdelivr.net/npm/onnxruntime-web@1.14.0/dist/ort.js\";\n document.head.appendChild(script1)\n const script2 = document.createElement(\"script\");\n script2.onload = async () => {\n console.log(\"vad loaded\");\n var record = document.querySelector('.record-button');\n record.textContent = \"Just Start Talking!\"\n \n const myvad = await vad.MicVAD.new({\n onSpeechStart: () => {\n var record = document.querySelector('.record-button');\n var player = document.querySelector('#streaming-out')\n if (record != null && (player == null || player.paused)) {\n record.click();\n }\n },\n onSpeechEnd: (audio) => {\n var stop = document.querySelector('.stop-button');\n if (stop != null) {\n stop.click();\n }\n }\n })\n myvad.start()\n }\n script2.src = \"https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.7/dist/bundle.min.js\";\n}\n```\n\nThis script loads our VAD model and sets up functions to start and stop recording automatically. When the user starts speaking, it triggers the recording, and when they stop, it ends the recording.\n\n---" }, { "id": 654, "parent": 649, "path": "07_streaming/06_automatic-voice-detection.md", "level": 3, "title": "Building a User Interface with Gradio", "content": "Now, let’s create an intuitive and visually appealing user interface with Gradio. This interface will include an audio input for capturing voice, a chat window for displaying responses, and state management to keep things synchronized.\n\n```python\nwith gr.Blocks(theme=theme, js=js) as demo:\n with gr.Row():\n input_audio = gr.Audio(\n label=\"Input Audio\",\n sources=[\"microphone\"],\n type=\"numpy\",\n streaming=False,\n waveform_options=gr.WaveformOptions(waveform_color=\"#B83A4B\"),\n )\n with gr.Row():\n chatbot = gr.Chatbot(label=\"Conversation\", type=\"messages\")\n state = gr.State(value=AppState())\n```\n\nIn this code block, we’re using Gradio’s `Blocks` API to create an interface with an audio input, a chat display, and an application state manager. The color customization for the waveform adds a nice visual touch.\n\n---" }, { "id": 655, "parent": 649, "path": "07_streaming/06_automatic-voice-detection.md", "level": 3, "title": "Handling Recording and Responses", "content": "Finally, let’s link the recording and response components to ensure the app reacts smoothly to user inputs and provides responses in real-time.\n\n```python\n stream = input_audio.start_recording(\n process_audio,\n [input_audio, state],\n [input_audio, state],\n )\n respond = input_audio.stop_recording(\n response, [state, input_audio], [state, chatbot]\n )\n```\n\nThese lines set up event listeners for starting and stopping the recording, processing the audio input, and generating responses. By linking these events, we create a cohesive experience where users can simply talk, and the chatbot handles the rest.\n\n---" }, { "id": 656, "parent": 649, "path": "07_streaming/06_automatic-voice-detection.md", "level": 2, "title": "Summary", "content": "1. When you open the app, the VAD system automatically initializes and starts listening for speech\n2. As soon as you start talking, it triggers the recording automatically\n3. When you stop speaking, the recording ends and:\n - The audio is transcribed using Whisper\n - The transcribed text is sent to the LLM\n - The LLM generates a response about calorie tracking\n - The response is displayed in the chat interface\n4. This creates a natural back-and-forth conversation where you can simply talk about your meals and get instant feedback on nutritional content\n\nThis app demonstrates how to create a natural voice interface that feels responsive and intuitive. By combining Groq's fast inference with automatic speech detection, we've eliminated the need for manual recording controls while maintaining high-quality interactions. The result is a practical calorie tracking assistant that users can simply talk to as naturally as they would to a human nutritionist.\n\nLink to GitHub repository: [Groq Gradio Basics](https://github.com/bklieger-groq/gradio-groq-basics/tree/main/calorie-tracker)" } ]