Flagging

You may have noticed the "Flag" button that appears by default in your Interface. When a user using your demo sees input with interesting output, such as erroneous or unexpected model behaviour, they can flag the input for you to review. Within the directory provided by the flagging_dir= argument to the Interface constructor, a CSV file will log the flagged inputs. If the interface involves file data, such as for Image and Audio components, folders will be created to store those flagged data as well.

For example, with the calculator interface shown above, we would have the flagged data stored in the flagged directory shown below:

+-- calculator.py
+-- flagged/
|   +-- logs.csv

flagged/logs.csv

num1,operation,num2,Output
5,add,7,12
6,subtract,1.5,4.5

With the sepia interface shown earlier, we would have the flagged data stored in the flagged directory shown below:

+-- sepia.py
+-- flagged/
|   +-- logs.csv
|   +-- im/
|   |   +-- 0.png
|   |   +-- 1.png
|   +-- Output/
|   |   +-- 0.png
|   |   +-- 1.png

flagged/logs.csv

im,Output
im/0.png,Output/0.png
im/1.png,Output/1.png

If you wish for the user to provide a reason for flagging, you can pass a list of strings to the flagging_options argument of Interface. Users will have to select one of the strings when flagging, which will be saved as an additional column to the CSV.