Add two cultural evals

app.py

@@ -141,8 +141,41 @@ The following categories are high-level, non-exhaustive, and present a synthesis
 
 
         with gr.TabItem("Cultural Values/Sensitive Content"):
+            fulltable = globaldf[globaldf['Group'] == 'CulturalEvals']
+            fulltable = fulltable[['Modality','Type', 'Suggested Evaluation', 'What it is evaluating', 'Considerations', 'Link']]
+
+            gr.Markdown("""Cultural values are specific to groups and sensitive content is normative. Sensitive topics also vary by culture and can include hate speech. What is considered a sensitive topic, such as egregious violence or adult sexual content, can vary widely by viewpoint. Due to norms differing by culture, region, and language, there is no standard for what constitutes sensitive content.
+                        Distinct cultural values present a challenge for deploying models into a global sphere, as what may be appropriate in one culture may be unsafe in others. Generative AI systems cannot be neutral or objective, nor can they encompass truly universal values. There is no “view from nowhere”; in quantifying anything, a particular frame of reference is imposed.
+                        """)
             with gr.Row():
-                gr.Image()
+                modality_filter = gr.CheckboxGroup(["Text", "Image", "Audio", "Video"], 
+                                                 value=["Text", "Image", "Audio", "Video"], 
+                                                 label="Modality", 
+                                                 show_label=True,
+                                                #  info="Which modality to show."
+                                                 )
+                type_filter = gr.CheckboxGroup(["Model", "Dataset", "Output", "Taxonomy"], 
+                                                 value=["Model", "Dataset", "Output", "Taxonomy"], 
+                                                 label="Type", 
+                                                 show_label=True,
+                                                #  info="Which modality to show."
+                                                 )
+            with gr.Row():
+                biastable_full = gr.DataFrame(value=fulltable, wrap=True, datatype="markdown", visible=False, interactive=False)
+                biastable_filtered = gr.DataFrame(value=fulltable, wrap=True, datatype="markdown", visible=True, interactive=False)
+                modality_filter.change(filter_modality, inputs=[biastable_filtered, modality_filter], outputs=biastable_filtered)
+                type_filter.change(filter_type, inputs=[biastable_filtered, type_filter], outputs=biastable_filtered)
+
+
+                with Modal(visible=False) as modal:
+                    titlemd = gr.Markdown(visible=False)
+                    authormd = gr.Markdown(visible=False)
+                    tagsmd = gr.Markdown(visible=False)
+                    abstractmd = gr.Markdown(visible=False)
+                    datasetmd = gr.Markdown(visible=False)
+                    gallery = gr.Gallery(visible=False)
+                biastable_filtered.select(showmodal, None, [modal, titlemd, authormd, tagsmd, abstractmd, datasetmd, gallery])
+
 
         # with gr.TabItem("Disparate Performance"):
         #     with gr.Row():

configs/palms.yaml

@@ -0,0 +1,14 @@
+Abstract: "Language models can generate harmful and biased outputs and exhibit undesirable behavior according to a given cultural context. We propose a Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets, an iterative process to significantly change model behavior by crafting and fine-tuning on a dataset that reflects a predetermined set of target values. We evaluate our process using three metrics: quantitative metrics with human evaluations that score output adherence to a target value, toxicity scoring on outputs; and qualitative metrics analyzing the most common word associated with a given social category. Through each iteration, we add additional training dataset examples based on observed shortcomings from evaluations. PALMS performs significantly better on all metrics compared to baseline and control models for a broad range of GPT-3 language model sizes without compromising capability integrity. We find that the effectiveness of PALMS increases with model size. We show that significantly adjusting language model behavior is feasible with a small, hand-curated dataset."
+Applicable Models: .nan
+Authors: Irene Solaiman, Christy Dennison
+Considerations: Requires predefining what adherence to a culture means for human evals
+Datasets: .nan
+Group: CulturalEvals
+Hashtags: .nan
+Link: 'Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets'
+Modality: Text
+Screenshots: .nan
+Suggested Evaluation: Human and Toxicity Evals of Cultural Value Categories
+Type: Output
+URL: http://arxiv.org/abs/2106.10328
+What it is evaluating: Adherence to defined norms for a set of cultural categories

configs/tango.yaml

@@ -0,0 +1,19 @@
+Abstract: "Transgender and non-binary (TGNB) individuals disproportionately experience discrimination and exclusion from daily life. Given the recent popularity and adoption of language generation technologies, the potential to further marginalize this population only grows. Although a multitude of NLP fairness literature focuses on illuminating and addressing gender biases, assessing gender harms for TGNB identities requires understanding how such identities uniquely interact with societal gender norms and how they differ from gender binary-centric perspectives. Such measurement frameworks inherently require centering TGNB voices to help guide the alignment between gender-inclusive NLP and whom they are intended to serve. Towards this goal, we ground our work in the TGNB community and existing interdisciplinary literature to assess how the social reality surrounding experienced marginalization of TGNB persons contributes to and persists within Open Language Generation (OLG). This social knowledge serves as a guide for evaluating popular large language models (LLMs) on two key aspects: (1) misgendering and (2) harmful responses to gender disclosure. To do this, we introduce TANGO, a dataset of template-based real-world text curated from a TGNB-oriented community. We discover a dominance of binary gender norms reflected by the models; LLMs least misgendered subjects in generated text when triggered by prompts whose subjects used binary pronouns. Meanwhile, misgendering was most prevalent when triggering generation with singular they and neopronouns. When prompted with gender disclosures, TGNB disclosure generated the most stigmatizing language and scored most toxic, on average. Our findings warrant further research on how TGNB harms manifest in LLMs and serve as a broader case study toward concretely grounding the design of gender-inclusive AI in community voices and interdisciplinary literature."
+Applicable Models:
+- GPT-2
+- GPT-Neo
+- OPT
+Authors: Anaelia Ovalle, Palash Goyal, Jwala Dhamala, Zachary Jaggers, Kai-Wei Chang, Aram Galstyan, Richard Zemel, Rahul Gupta
+Considerations: Based on automatic evaluations of the resulting open language generation, may be sensitive to the choice of evaluator. Would advice for a combination of perspective, detoxify, and regard metrics
+Datasets: https://huggingface.co/datasets/AlexaAI/TANGO
+Group: CulturalEvals
+Hashtags: .nan
+Link: '“I’m fully who I am”: Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation'
+Modality: Text
+Screenshots:
+- Images/TANGO1.png
+- Images/TANGO2.png
+Suggested Evaluation: Human and Toxicity Evals of Cultural Value Categories
+Type: Output
+URL: http://arxiv.org/abs/2106.10328
+What it is evaluating: Bias measurement for trans and nonbinary community via measuring gender non-affirmative language, specifically 1) misgendering 2), negative responses to gender disclosure