{ "cells": [ { "cell_type": "markdown", "id": "a51ae982-cad6-4d26-a18a-83472879325b", "metadata": {}, "source": [ "# Setup" ] }, { "cell_type": "markdown", "id": "538e8d9b-74c8-4684-80be-623778bdaa90", "metadata": {}, "source": [ "## Config" ] }, { "cell_type": "code", "execution_count": 1, "id": "cf368153-eb55-42f4-887f-889c4e9d7c60", "metadata": {}, "outputs": [], "source": [ "INPUT_DATASET = 'layoric/labeled-multiple-choice-explained'\n", "REVISION = '536f3b8'\n", "OUTPUT_DATASET = 'derek-thomas/labeled-multiple-choice-explained-mistral-reasoning'\n", "NAMESPACE = 'HF-test-lab' # Use your username or wherever you have the ability to deploy inference endpoints" ] }, { "cell_type": "code", "execution_count": 2, "id": "c5a12bab-97bf-4a1d-989b-89f50a7fc272", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "aed670be03f741699118c33316dce7ec", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='
\n", " | formatted_question | \n", "combinedfact | \n", "answerKey | \n", "topic | \n", "__index_level_0__ | \n", "explanation | \n", "
---|---|---|---|---|---|---|
0 | \n", "what is satellite technology used for predicti... | \n", "satellite technology is used for predicting wh... | \n", "c | \n", "technology | \n", "35972.0 | \n", "a) Seconds and minutes: This option is incorre... | \n", "
1 | \n", "what does irradiating food do? (a) relieve pai... | \n", "irradiated food improves food safety. | \n", "c | \n", "food science | \n", "20412.0 | \n", "(a) Relieve pain: This option is not correct b... | \n", "
2 | \n", "what protects a mammal's skin? (a) fiber folli... | \n", "fiber follicles protect mammal skin | \n", "a | \n", "biology | \n", "5779.0 | \n", "b) Exfoliation: Exfoliation is the process of ... | \n", "
3 | \n", "what do earthworms do when a segment breaks of... | \n", "earthworms can regrow segments that break off | \n", "b | \n", "biology | \n", "4064.0 | \n", "a) Dies: This option is not correct because ea... | \n", "
4 | \n", "lightning can be bad for what? (a) the environ... | \n", "lightning can be bad for the environment. | \n", "a | \n", "electricity | \n", "34080.0 | \n", "b) Rainstorms: Lightning is actually a natural... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
9093 | \n", "organisms that can cause infection do what? (a... | \n", "organisms that can cause infection make humans... | \n", "g | \n", "biology | \n", "33485.0 | \n", "a) Bandaging open sores is not the correct ans... | \n", "
9094 | \n", "fungi are living things that cannot make thei... | \n", "fungi are living things that cannot make their... | \n", "a | \n", "biology | \n", "12097.0 | \n", "b) Fungi are living things that can make their... | \n", "
9095 | \n", "an overheated body can use water for: (a) meta... | \n", "the evaporation of water from the skin cools t... | \n", "g | \n", "biology | \n", "6522.0 | \n", "a) Metabolic reaction: This option is incorrec... | \n", "
9096 | \n", "what is essential for cellular respiration for... | \n", "plants are essential for cellular respiration ... | \n", "f | \n", "biology | \n", "27144.0 | \n", "a) Electrons are involved in cellular respirat... | \n", "
9097 | \n", "what helps insulate and protect the body? (a) ... | \n", "living cells in follicles help insulate and pr... | \n", "b | \n", "biology | \n", "18522.0 | \n", "a) H2O: Water is essential for life, but it do... | \n", "
9098 rows × 6 columns
\n", "" ], "text/plain": [ " formatted_question \\\n", "0 what is satellite technology used for predicti... \n", "1 what does irradiating food do? (a) relieve pai... \n", "2 what protects a mammal's skin? (a) fiber folli... \n", "3 what do earthworms do when a segment breaks of... \n", "4 lightning can be bad for what? (a) the environ... \n", "... ... \n", "9093 organisms that can cause infection do what? (a... \n", "9094 fungi are living things that cannot make thei... \n", "9095 an overheated body can use water for: (a) meta... \n", "9096 what is essential for cellular respiration for... \n", "9097 what helps insulate and protect the body? (a) ... \n", "\n", " combinedfact answerKey \\\n", "0 satellite technology is used for predicting wh... c \n", "1 irradiated food improves food safety. c \n", "2 fiber follicles protect mammal skin a \n", "3 earthworms can regrow segments that break off b \n", "4 lightning can be bad for the environment. a \n", "... ... ... \n", "9093 organisms that can cause infection make humans... g \n", "9094 fungi are living things that cannot make their... a \n", "9095 the evaporation of water from the skin cools t... g \n", "9096 plants are essential for cellular respiration ... f \n", "9097 living cells in follicles help insulate and pr... b \n", "\n", " topic __index_level_0__ \\\n", "0 technology 35972.0 \n", "1 food science 20412.0 \n", "2 biology 5779.0 \n", "3 biology 4064.0 \n", "4 electricity 34080.0 \n", "... ... ... \n", "9093 biology 33485.0 \n", "9094 biology 12097.0 \n", "9095 biology 6522.0 \n", "9096 biology 27144.0 \n", "9097 biology 18522.0 \n", "\n", " explanation \n", "0 a) Seconds and minutes: This option is incorre... \n", "1 (a) Relieve pain: This option is not correct b... \n", "2 b) Exfoliation: Exfoliation is the process of ... \n", "3 a) Dies: This option is not correct because ea... \n", "4 b) Rainstorms: Lightning is actually a natural... \n", "... ... \n", "9093 a) Bandaging open sores is not the correct ans... \n", "9094 b) Fungi are living things that can make their... \n", "9095 a) Metabolic reaction: This option is incorrec... \n", "9096 a) Electrons are involved in cellular respirat... \n", "9097 a) H2O: Water is essential for life, but it do... \n", "\n", "[9098 rows x 6 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "code", "execution_count": 7, "id": "a1ec5678-13a5-4602-9d67-8406248fa414", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Before Cleaning: 9098 rows\n", "After Cleaning: 8413 rows\n" ] } ], "source": [ "print(f\"Before Cleaning: {len(df)} rows\")\n", "\n", "# Drop the __index_level_0__ column if it exists\n", "df.drop(columns=['__index_level_0__'], errors='ignore', inplace=True)\n", "\n", "# Ensure all values in 'formatted_question' are strings\n", "df['formatted_question'] = df['formatted_question'].astype(str)\n", "\n", "# Filter on topic so that we can stratify later\n", "df['topic'] = df['topic'].fillna('unknown')\n", "topic_counts = df['topic'].value_counts()\n", "valid_topics = topic_counts[topic_counts >= 5].index\n", "df = df[df['topic'].isin(valid_topics)]\n", "df = df[~df['topic'].isin(['unknown'])]\n", "\n", "df.rename(columns={\n", " 'combinedfact': 'combined_fact',\n", " 'answerKey': 'answer_key' \n", "}, inplace=True)\n", "\n", "\n", "# Drop the index level 0 if it exists\n", "df.reset_index(drop=True, inplace=True)\n", "print(f\"After Cleaning: {len(df)} rows\")" ] }, { "cell_type": "code", "execution_count": 8, "id": "71337823-91b9-4946-95cd-71810f5c5101", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " | formatted_question | \n", "combinedfact | \n", "answer_key | \n", "topic | \n", "explanation | \n", "
---|---|---|---|---|---|
0 | \n", "what is satellite technology used for predicti... | \n", "satellite technology is used for predicting wh... | \n", "c | \n", "technology | \n", "a) Seconds and minutes: This option is incorre... | \n", "
1 | \n", "what does irradiating food do? (a) relieve pai... | \n", "irradiated food improves food safety. | \n", "c | \n", "food science | \n", "(a) Relieve pain: This option is not correct b... | \n", "
2 | \n", "what protects a mammal's skin? (a) fiber folli... | \n", "fiber follicles protect mammal skin | \n", "a | \n", "biology | \n", "b) Exfoliation: Exfoliation is the process of ... | \n", "
3 | \n", "what do earthworms do when a segment breaks of... | \n", "earthworms can regrow segments that break off | \n", "b | \n", "biology | \n", "a) Dies: This option is not correct because ea... | \n", "
4 | \n", "lightning can be bad for what? (a) the environ... | \n", "lightning can be bad for the environment. | \n", "a | \n", "electricity | \n", "b) Rainstorms: Lightning is actually a natural... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
8408 | \n", "organisms that can cause infection do what? (a... | \n", "organisms that can cause infection make humans... | \n", "g | \n", "biology | \n", "a) Bandaging open sores is not the correct ans... | \n", "
8409 | \n", "fungi are living things that cannot make thei... | \n", "fungi are living things that cannot make their... | \n", "a | \n", "biology | \n", "b) Fungi are living things that can make their... | \n", "
8410 | \n", "an overheated body can use water for: (a) meta... | \n", "the evaporation of water from the skin cools t... | \n", "g | \n", "biology | \n", "a) Metabolic reaction: This option is incorrec... | \n", "
8411 | \n", "what is essential for cellular respiration for... | \n", "plants are essential for cellular respiration ... | \n", "f | \n", "biology | \n", "a) Electrons are involved in cellular respirat... | \n", "
8412 | \n", "what helps insulate and protect the body? (a) ... | \n", "living cells in follicles help insulate and pr... | \n", "b | \n", "biology | \n", "a) H2O: Water is essential for life, but it do... | \n", "
8413 rows × 5 columns
\n", "\n", " | formatted_question | \n", "combinedfact | \n", "answer_key | \n", "topic | \n", "explanation | \n", "question_text | \n", "answer_choices | \n", "
---|---|---|---|---|---|---|---|
0 | \n", "what is satellite technology used for predicti... | \n", "satellite technology is used for predicting wh... | \n", "c | \n", "Technology | \n", "a) Seconds and minutes: This option is incorre... | \n", "What is satellite technology used for predicting? | \n", "(a) Seconds and minutes (b) The strength and m... | \n", "
1 | \n", "what does irradiating food do? (a) relieve pai... | \n", "irradiated food improves food safety. | \n", "c | \n", "Food science | \n", "(a) Relieve pain: This option is not correct b... | \n", "What does irradiating food do? | \n", "(a) Relieve pain (b) Enhance food's nutrients ... | \n", "
2 | \n", "what protects a mammal's skin? (a) fiber folli... | \n", "fiber follicles protect mammal skin | \n", "a | \n", "Biology | \n", "b) Exfoliation: Exfoliation is the process of ... | \n", "What protects a mammal's skin? | \n", "(a) Fiber follicles (b) Exfoliation (c) Resist... | \n", "
3 | \n", "what do earthworms do when a segment breaks of... | \n", "earthworms can regrow segments that break off | \n", "b | \n", "Biology | \n", "a) Dies: This option is not correct because ea... | \n", "What do earthworms do when a segment breaks off? | \n", "(a) Dies (b) Regrows it (c) Reproduces (d) Sed... | \n", "
4 | \n", "lightning can be bad for what? (a) the environ... | \n", "lightning can be bad for the environment. | \n", "a | \n", "Electricity | \n", "b) Rainstorms: Lightning is actually a natural... | \n", "Lightning can be bad for what? | \n", "(a) The environment (b) Rainstorms (c) Destruc... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
8408 | \n", "organisms that can cause infection do what? (a... | \n", "organisms that can cause infection make humans... | \n", "g | \n", "Biology | \n", "a) Bandaging open sores is not the correct ans... | \n", "Organisms that can cause infection do what? | \n", "(a) Bandage open sores (b) Keep flesh clean (c... | \n", "
8409 | \n", "fungi are living things that cannot make thei... | \n", "fungi are living things that cannot make their... | \n", "a | \n", "Biology | \n", "b) Fungi are living things that can make their... | \n", "Fungi are living things that cannot make their... | \n", "(a) Food (b) Cells (c) Energy (d) Fruits (e) H... | \n", "
8410 | \n", "an overheated body can use water for: (a) meta... | \n", "the evaporation of water from the skin cools t... | \n", "g | \n", "Biology | \n", "a) Metabolic reaction: This option is incorrec... | \n", "An overheated body can use water for:? | \n", "(a) Metabolic reaction (b) Dehydrating (c) Rai... | \n", "
8411 | \n", "what is essential for cellular respiration for... | \n", "plants are essential for cellular respiration ... | \n", "f | \n", "Biology | \n", "a) Electrons are involved in cellular respirat... | \n", "What is essential for cellular respiration for... | \n", "(a) Electron (b) Glucose (c) Energy (d) Energy... | \n", "
8412 | \n", "what helps insulate and protect the body? (a) ... | \n", "living cells in follicles help insulate and pr... | \n", "b | \n", "Biology | \n", "a) H2O: Water is essential for life, but it do... | \n", "What helps insulate and protect the body? | \n", "(a) H2o (b) Living cells in follicles (c) Laye... | \n", "
8413 rows × 7 columns
\n", "\n", " | formatted_question | \n", "combined_fact | \n", "answer_key | \n", "topic | \n", "explanation | \n", "question_text | \n", "answer_choices | \n", "mistral_reasoning_prompt | \n", "mistral_reasoning | \n", "
---|---|---|---|---|---|---|---|---|---|
0 | \n", "what is satellite technology used for predicti... | \n", "satellite technology is used for predicting wh... | \n", "c | \n", "Technology | \n", "a) Seconds and minutes: This option is incorre... | \n", "What is satellite technology used for predicting? | \n", "(a) Seconds and minutes (b) The strength and m... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "Incorrect answers and explanations:\\n\\n1. Elec... | \n", "
1 | \n", "what does irradiating food do? (a) relieve pai... | \n", "irradiated food improves food safety. | \n", "c | \n", "Food science | \n", "(a) Relieve pain: This option is not correct b... | \n", "What does irradiating food do? | \n", "(a) Relieve pain (b) Enhance food's nutrients ... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "Sure, let's examine each answer and justify wh... | \n", "
2 | \n", "what protects a mammal's skin? (a) fiber folli... | \n", "fiber follicles protect mammal skin | \n", "a | \n", "Biology | \n", "b) Exfoliation: Exfoliation is the process of ... | \n", "What protects a mammal's skin? | \n", "(a) Fiber follicles (b) Exfoliation (c) Resist... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "Sure, let's go through each of the provided an... | \n", "
3 | \n", "what do earthworms do when a segment breaks of... | \n", "earthworms can regrow segments that break off | \n", "b | \n", "Biology | \n", "a) Dies: This option is not correct because ea... | \n", "What do earthworms do when a segment breaks off? | \n", "(a) Dies (b) Regrows it (c) Reproduces (d) Sed... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "1. Reading the question carefully, we can see ... | \n", "
4 | \n", "lightning can be bad for what? (a) the environ... | \n", "lightning can be bad for the environment. | \n", "a | \n", "Electricity | \n", "b) Rainstorms: Lightning is actually a natural... | \n", "Lightning can be bad for what? | \n", "(a) The environment (b) Rainstorms (c) Destruc... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "1. Food: While essential for the growth and he... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
8408 | \n", "organisms that can cause infection do what? (a... | \n", "organisms that can cause infection make humans... | \n", "g | \n", "Biology | \n", "a) Bandaging open sores is not the correct ans... | \n", "Organisms that can cause infection do what? | \n", "(a) Bandage open sores (b) Keep flesh clean (c... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "1. Read the question and options carefully: Th... | \n", "
8409 | \n", "fungi are living things that cannot make thei... | \n", "fungi are living things that cannot make their... | \n", "a | \n", "Biology | \n", "b) Fungi are living things that can make their... | \n", "Fungi are living things that cannot make their... | \n", "(a) Food (b) Cells (c) Energy (d) Fruits (e) H... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "1. Read the question and options carefully: Th... | \n", "
8410 | \n", "an overheated body can use water for: (a) meta... | \n", "the evaporation of water from the skin cools t... | \n", "g | \n", "Biology | \n", "a) Metabolic reaction: This option is incorrec... | \n", "An overheated body can use water for:? | \n", "(a) Metabolic reaction (b) Dehydrating (c) Rai... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "1. Read the question and options carefully: Th... | \n", "
8411 | \n", "what is essential for cellular respiration for... | \n", "plants are essential for cellular respiration ... | \n", "f | \n", "Biology | \n", "a) Electrons are involved in cellular respirat... | \n", "What is essential for cellular respiration for... | \n", "(a) Electron (b) Glucose (c) Energy (d) Energy... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "1. First, let's read the question and options ... | \n", "
8412 | \n", "what helps insulate and protect the body? (a) ... | \n", "living cells in follicles help insulate and pr... | \n", "b | \n", "Biology | \n", "a) H2O: Water is essential for life, but it do... | \n", "What helps insulate and protect the body? | \n", "(a) H2o (b) Living cells in follicles (c) Laye... | \n", "[{'role': 'user', 'content': 'You are an AI as... | \n", "1. Read the question and options carefully: Th... | \n", "
8413 rows × 9 columns
\n", "