prompt_with_context.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_context.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LLMs for Self-Study\n",
    "> A prompt and code template for better understanding texts\n",
    "\n",
    "This notebook provides a guide for using LLMs for self-study programmatically. A number of prompt templates are provided to assist with generating great assessments for self-study, and code is additionally provided for fast usage. This notebook is best leveraged for a set of documents (text or PDF preferred) **to be uploaded** for interaction with the model.\n",
    "\n",
    "This version of the notebook is best suited for those who prefer to use files from their local drive as context rather than copy and pasting directly into the notebook to be used as context for the model. If you prefer to copy and paste text, you should direct yourself to the [prompt_with_context](https://colab.research.google.com/github/vanderbilt-data-science/lo-achievement/blob/main/prompt_with_context.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code Setup\n",
    "Run the following cells to setup the rest of the environment for prompting. In the following section, we set up the computational environment with imported code, setup your API key access to OpenAI, and loading access to your language model. Note that the following cells may take a long time to run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Library installation and loading\n",
    "The following `pip install` code should be run if you're using Google Colab, or otherwise do not have a computational environment (e.g., _venv_, _conda virtual environment_, _Docker, Singularity, or other container_) with these packages installed."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "skip_exec: true\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this code if you're using Google Colab or don't have these packages installed in your computing environment\n",
    "! pip install pip install git+https://<token>@github.com/vanderbilt-data-science/lo-achievement.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "import numpy as np\n",
    "import getpass\n",
    "import os\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.chains import RetrievalQA\n",
    "from langchain.schema import SystemMessage, HumanMessage, AIMessage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# libraries from our package\n",
    "from ai_classroom_suite.PromptInteractionBase import *\n",
    "import ai_classroom_suite.SelfStudyPrompts as ssp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API and model setup\n",
    "\n",
    "Use these cells to load the API keys required for this notebook and create a basic OpenAI LLM model. The code below uses the variable you created above when you input your API Key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up OpenAI API Key\n",
    "set_openai_key()\n",
    "\n",
    "# Create model\n",
    "mdl = create_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inspect Available Default Prompts\n",
    "A number of default prompts have been provided for you so you don't need to form your own prompt to begin with. They will be listed below, and the different ways to interact with them are displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SELF_STUDY_DEFAULTS key</th>\n",
       "      <th>Prompt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mc</td>\n",
       "      <td>Please design a 5 question multiple choice quiz about the provided text.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>short_answer</td>\n",
       "      <td>Please design a 5 question short answer quiz about the provided text. The question types should be short answer. Expect the correct answers to be a few sentences long.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fill_blank</td>\n",
       "      <td>Create a 5 question fill in the blank quiz referencing parts of the provided text.\\nThe \"blank\" part of the question should appear as \"________\". The answers should reflect what word(s) should go in the blank an accurate statement.\\nAn example is as follows: \"The author of the book is ______.\" The question should be a statement.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sequencing</td>\n",
       "      <td>Create a 5 question questionnaire that will ask me to recall the steps or sequence of events\\nin the provided text.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>relationships</td>\n",
       "      <td>Create a 5 question quiz for the student that asks the student to identify relationships betweentopics or concepts that are important to understanding this text.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>concepts</td>\n",
       "      <td>Design a 5 question quiz that asks me about definitions or concepts of importance in the provided text.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>real_world_example</td>\n",
       "      <td>Demonstrate how the provided context can be applied to solve a real world problem.\\nAsk me questions about how the demonstration you provided relates to solving a real world problem.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>randomized_questions</td>\n",
       "      <td>Generate a high-quality assessment consisting of 5 varied questions,\\neach of different types (open-ended, multiple choice, short answer, analogies, etc.)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  SELF_STUDY_DEFAULTS key  \\\n",
       "0                      mc   \n",
       "1            short_answer   \n",
       "2              fill_blank   \n",
       "3              sequencing   \n",
       "4           relationships   \n",
       "5                concepts   \n",
       "6      real_world_example   \n",
       "7    randomized_questions   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                         Prompt  \n",
       "0                                                                                                                                                                                                                                                                      Please design a 5 question multiple choice quiz about the provided text.  \n",
       "1                                                                                                                                                                       Please design a 5 question short answer quiz about the provided text. The question types should be short answer. Expect the correct answers to be a few sentences long.  \n",
       "2  Create a 5 question fill in the blank quiz referencing parts of the provided text.\\nThe \"blank\" part of the question should appear as \"________\". The answers should reflect what word(s) should go in the blank an accurate statement.\\nAn example is as follows: \"The author of the book is ______.\" The question should be a statement.\\n  \n",
       "3                                                                                                                                                                                                                           Create a 5 question questionnaire that will ask me to recall the steps or sequence of events\\nin the provided text.  \n",
       "4                                                                                                                                                                             Create a 5 question quiz for the student that asks the student to identify relationships betweentopics or concepts that are important to understanding this text.  \n",
       "5                                                                                                                                                                                                                                       Design a 5 question quiz that asks me about definitions or concepts of importance in the provided text.  \n",
       "6                                                                                                                                                        Demonstrate how the provided context can be applied to solve a real world problem.\\nAsk me questions about how the demonstration you provided relates to solving a real world problem.  \n",
       "7                                                                                                                                                                                    Generate a high-quality assessment consisting of 5 varied questions,\\neach of different types (open-ended, multiple choice, short answer, analogies, etc.)  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show all prompts and names\n",
    "ssp.print_all_self_study_prompts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Please design a 5 question multiple choice quiz about the provided text.'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# accessing texts of desired assessment types\n",
    "ssp.SELF_STUDY_DEFAULTS['mc']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Add your context and assign the prefix to your query.\n",
    "The query assigned here serves as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\" Two roads diverged in a yellow wood,\n",
    "And sorry I could not travel both\n",
    "And be one traveler, long I stood\n",
    "And looked down one as far as I could\n",
    "To where it bent in the undergrowth;\n",
    "Then took the other, as just as fair,\n",
    "And having perhaps the better claim,\n",
    "Because it was grassy and wanted wear;\n",
    "Though as for that the passing there\n",
    "Had worn them really about the same,\n",
    "And both that morning equally lay\n",
    "In leaves no step had trodden black.\n",
    "Oh, I kept the first for another day!\n",
    "Yet knowing how way leads on to way,\n",
    "I doubted if I should ever come back.\n",
    "I shall be telling this with a sigh\n",
    "Somewhere ages and ages hence:\n",
    "Two roads diverged in a wood, and I—\n",
    "I took the one less traveled by,\n",
    "And that has made all the difference.\n",
    "—-Robert Frost—-\n",
    "Education Place: http://www.eduplace.com \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A guide to prompting for self-study\n",
    "In this section, we provide a number of different approaches for using AI to help you assess and explain the knowledge of your document. Start by interacting with the model and then try out the rest of the prompts!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interact with the model\n",
    "\n",
    "Now that your context is created, you can begin interacting with the model! Below, we have a comprehensive list of examples using different question types, but feel free to use this code block to experiment with the model.\n",
    "\n",
    "First, let's make the settings for the query. In other words, what are the learning objectives and what is the type of assessment we want to have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set short answer as the assessment type\n",
    "assessment_type = ssp.SELF_STUDY_DEFAULTS[\"short_answer\"]\n",
    "\n",
    "# set learning objectives if desired\n",
    "learning_objs = (\"\"\"1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
    "                 2. Understand the literary devices used in poetry and their purposes.\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's use the predefined defaults with the model and provided APIs from `ai_classroom_suite`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Question 1: Who is the narrator of the poem and what is the setting?\\n\\nPlease provide your answer in a few sentences.'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ask the tutor to prompt you based on the text\n",
    "get_tutoring_answer(context, mdl, assessment_request = assessment_type,\n",
    "                    learning_objectives = learning_objs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The complete prompt sent is based on pre-generated information and can be seen below. Your context, assessment_type, and learning objectives are substituted to create a full prompt as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System: You are a world-class tutor helping students to perform better on oral and written exams though interactive experiences. When assessing and evaluating students, you always ask one question at a time, and wait for the student's response before providing them with feedback. Asking one question at a time, waiting for the student's response, and then commenting on the strengths and weaknesses of their responses (when appropriate) is what makes you such a sought-after, world-class tutor.\n",
      "Human: I'm trying to better understand the text provided below. Please design a 5 question short answer quiz about the provided text. The question types should be short answer. Expect the correct answers to be a few sentences long. The learning objectives to be assessed are: 1. Identify the key elements of the poem: narrator, setting, and underlying message.\n",
      "                 2. Understand the literary devices used in poetry and their purposes.. Although I may request more than one assessment question, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right. The text that you will base your questions on is as follows:  Two roads diverged in a yellow wood,\n",
      "And sorry I could not travel both\n",
      "And be one traveler, long I stood\n",
      "And looked down one as far as I could\n",
      "To where it bent in the undergrowth;\n",
      "Then took the other, as just as fair,\n",
      "And having perhaps the better claim,\n",
      "Because it was grassy and wanted wear;\n",
      "Though as for that the passing there\n",
      "Had worn them really about the same,\n",
      "And both that morning equally lay\n",
      "In leaves no step had trodden black.\n",
      "Oh, I kept the first for another day!\n",
      "Yet knowing how way leads on to way,\n",
      "I doubted if I should ever come back.\n",
      "I shall be telling this with a sigh\n",
      "Somewhere ages and ages hence:\n",
      "Two roads diverged in a wood, and I—\n",
      "I took the one less traveled by,\n",
      "And that has made all the difference.\n",
      "—-Robert Frost—-\n",
      "Education Place: http://www.eduplace.com .\n"
     ]
    }
   ],
   "source": [
    "# Use different function to create the prompt\n",
    "full_prompt = get_tutoring_prompt(context, assessment_request = assessment_type,\n",
    "                                  learning_objectives = learning_objs)\n",
    "\n",
    "# Show the prompt as a string\n",
    "print(full_prompt.to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternately, you can define your own prompt, which you'll as appropriate. To modify the kind of assessment you'll be asking for, change `assessment_request`. An example of how to add more context to the model is shown below as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Question: Who is the narrator in the poem and what is the underlying message conveyed?\\n\\nHint: Pay attention to the pronouns used throughout the poem to determine the narrator's identity. Additionally, think about the choices made by the narrator and the impact those choices have on their life. The underlying message is related to the consequences of these choices.\\n\\nTake your time to reflect on the text and provide your answer when you're ready.\""
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use your own texts\n",
    "custom_request = (\"Ask me a short answer question about the provided text. The questions you ask should allow\"\n",
    "                  \" me to demonstrate my creativity, capacity for out-of-the-box thinking, insights, and deeper meaning \"\n",
    "                  \" of the text.\")\n",
    "additional_context = (\"This is a text written by Robert Frost, a famous American poet. The text is widely studied in K-12 literature\"\n",
    "                    \" education courses, and should be read with an eye towards the philosophical and human themes of the text.\")\n",
    "\n",
    "# Concatenate context\n",
    "informed_context = context + \"\\n Additional information about the text is: \" + additional_context \n",
    "\n",
    "# Use custom_request defined as the assessment request\n",
    "get_tutoring_answer(informed_context, mdl, assessment_request = custom_request,\n",
    "                    learning_objectives = learning_objs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of Questions and Prompts\n",
    "\n",
    "Below is a comprehensive list of question types and prompt templates designed by our team. There are also example code blocks, where you can see how the model performed with the example and try it for yourself using the prompt template."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple Choice\n",
    "\n",
    "Prompt: The following text should be used as the basis for the instructions which follow: {context}. Please design a {number of questions} question quiz about {name or reference to context} which reflects the learning objectives: {list of learning objectives}. The questions should be multiple choice. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: Who is the narrator of the poem?\n",
      "\n",
      "A) Robert Frost\n",
      "B) The traveler \n",
      "C) The undergrowth \n",
      "D) The wood\n",
      "\n",
      "Please provide your answer.\n"
     ]
    }
   ],
   "source": [
    "# Multiple choice code example\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['mc'],\n",
    "                              learning_objectives = learning_objs)\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short Answer\n",
    "\n",
    "Prompt: Please design a {number of questions} question quiz about {context} which reflects the learning objectives: {list of learning objectives}. The questions should be short answer. Expect the correct answers to be {anticipated length} long. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: Who is the narrator of the poem and what is the setting?\n",
      "\n",
      "Remember to answer the question by identifying the narrator of the poem and describing the setting in which the events take place.\n"
     ]
    }
   ],
   "source": [
    "# Short answer code example\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['short_answer'],\n",
    "                              learning_objectives = learning_objs)\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fill-in-the-blank\n",
    "\n",
    "Prompt: Create a {number of questions} question fill in the blank quiz refrencing {context}. The quiz should reflect the learning objectives: {learning objectives}. The \"blank\" part of the question should appear as \"________\". The answers should reflect what word(s) should go in the blank an accurate statement.\n",
    "\n",
    "An example is the follow: \"The author of the book is \"________.\"\n",
    "\n",
    "The question should be a statement. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: The poem \"The Road Not Taken\" was written by ________.\n",
      "\n",
      "Question 2: What is the color of the wood where the two roads diverged? \n",
      "\n",
      "Question 3: What is the reason the narrator gives for choosing the second road?\n",
      "\n",
      "Question 4: What does the narrator say about the wear of both roads?\n",
      "\n",
      "Question 5: According to the poem, what has made all the difference in the narrator's life?\n",
      "\n",
      "Remember to wait for the student's response before providing feedback.\n"
     ]
    }
   ],
   "source": [
    "# Fill in the blank code example\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['fill_blank'],\n",
    "                              learning_objectives = learning_objs)\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequencing\n",
    "\n",
    "Prompt: Please develop a {number of questions} question questionnaire that will ask me to recall the steps involved in the following learning objectives in regard to {context}: {learning objectives}. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect, and then give me additional chances to respond until I get the correct choice. After I respond, explain their sequence to me."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: Who is the narrator of the poem?\n",
      "\n",
      "Question 2: What is the setting of the poem?\n",
      "\n",
      "Question 3: What is the underlying message of the poem?\n",
      "\n",
      "Question 4: What literary device is used in the line \"Two roads diverged in a yellow wood\"?\n",
      "\n",
      "Question 5: What is the purpose of using the literary device in question 4?\n",
      "\n",
      "Please answer question 1 first.\n"
     ]
    }
   ],
   "source": [
    "# Sequence example\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['sequencing'],\n",
    "                              learning_objectives = learning_objs)\n",
    "\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relationships/drawing connections\n",
    "\n",
    "Prompt: Please design a {number of questions} question quiz that asks me to explain the relationships that exist within the following learning objectives, referencing {context}: {learning objectives}. Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: Who is the narrator of the poem?\n",
      "\n",
      "Question 2: What is the setting of the poem?\n",
      "\n",
      "Question 3: What is the underlying message of the poem?\n",
      "\n",
      "Question 4: What literary device is used when the narrator says, \"Two roads diverged in a yellow wood\"?\n",
      "\n",
      "Question 5: What literary device is used when the narrator says, \"I took the one less traveled by, And that has made all the difference\"?\n"
     ]
    }
   ],
   "source": [
    "# Relationships example\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['relationships'],\n",
    "                              learning_objectives = learning_objs)\n",
    "\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concepts and Definitions\n",
    "\n",
    "Prompt: Design a {number of questions} question quiz that asks me about definitions related to the following learning objectives: {learning objectives} - based on {context}\".\n",
    "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: Who is the narrator of the poem? \n",
      "\n",
      "Remember, the narrator is the person who is speaking or telling the story.\n"
     ]
    }
   ],
   "source": [
    "# Concepts and definitions example\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['concepts'],\n",
    "                              learning_objectives = learning_objs)\n",
    "\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Real Word Examples\n",
    "\n",
    "Prompt: Demonstrate how {context} can be applied to solve a real-world problem related to the following learning objectives: {learning objectives}. Ask me questions regarding this theory/concept.\n",
    "\n",
    "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: Who is the narrator of the poem?\n",
      "\n",
      "Question 2: What is the setting of the poem?\n",
      "\n",
      "Question 3: What is the underlying message of the poem?\n",
      "\n",
      "Remember to provide your answer to one question at a time.\n"
     ]
    }
   ],
   "source": [
    "# Real word example\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['real_world_example'],\n",
    "                              learning_objectives = learning_objs)\n",
    "\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Randomized Question Types\n",
    "\n",
    "Prompt: Please generate a high-quality assessment consisting of {number of questions} varying questions, each of different types (open-ended, multiple choice, etc.), to determine if I achieved the following learning objectives in regards to {context}: {learning objectives}.\n",
    "\n",
    "Provide one question at a time, and wait for my response before providing me with feedback. Again, while the quiz may ask for multiple questions, you should only provide ONE question in you initial response. Do not include the answer in your response. If I get an answer wrong, provide me with an explanation of why it was incorrect,and then give me additional chances to respond until I get the correct choice. Explain why the correct choice is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1 (Open-ended): Who is the narrator of the poem and what is the setting?\n",
      "\n",
      "Question 2 (Multiple choice): Which literary device is used in the line \"And sorry I could not travel both\"?\n",
      "\n",
      "a) Simile\n",
      "b) Metaphor\n",
      "c) Alliteration\n",
      "d) Personification\n",
      "\n",
      "Question 3 (Short answer): Describe the underlying message of the poem in one sentence.\n",
      "\n",
      "Question 4 (Analogies): Complete the analogy: \"The two roads diverged in a yellow wood\" is to the physical setting as \"I took the one less traveled by\" is to ___________.\n",
      "\n",
      "Question 5 (Open-ended): Identify and explain one additional literary device used in the poem and its purpose.\n",
      "\n",
      "Please choose one question from above for me to provide a detailed evaluation.\n"
     ]
    }
   ],
   "source": [
    "# Randomized question types\n",
    "tutor_q = get_tutoring_answer(context, mdl, assessment_request = ssp.SELF_STUDY_DEFAULTS['randomized_questions'],\n",
    "                              learning_objectives = learning_objs)\n",
    "\n",
    "print(tutor_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantiative evaluation the correctness of a student's answer\n",
    "\n",
    "Prompt: (A continuation of the previous chat) Please generate the main points of the student’s answer to the previous question, and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives, and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the provided text, the student's answer to the previous question was not provided. Therefore, I cannot generate the main points of the student's answer or evaluate its comprehensiveness in relation to the learning objectives. Please provide the student's answer to the previous question so that I can assist you further.\n"
     ]
    }
   ],
   "source": [
    "# qualitative evaluation\n",
    "qualitative_query = \"\"\" Please generate the main points of the student’s answer to the previous question,\n",
    " and evaluate on a scale of 1 to 5 how comprehensive the student’s answer was in relation to the learning objectives,\n",
    " and explain why he or she received this rating, including what was missed in his or her answer if the student’s answer wasn’t complete.\"\"\"\n",
    "\n",
    "# Note that this uses the previous result and query in the context\n",
    "last_answer = ''\n",
    "\n",
    "# Get result with formatting to emphasize changes in parameter inputs\n",
    "result = get_tutoring_answer(last_answer + context,\n",
    "                             mdl,\n",
    "                             assessment_request = qualitative_query,\n",
    "                             learning_objectives = learning_objs)\n",
    "\n",
    "print(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}