Merge pull request #2 from basitanwer/first_commit

.gitignore

@@ -0,0 +1,4 @@
+*.tar.gz
+*.pyc
+results/
+models/

.gitignore/.gitignore

@@ -1 +0,0 @@
-foo.bar

README.md

@@ -0,0 +1,161 @@
+<!--lint disable no-literal-urls-->
+<p align="center">
+  <a href="https://zir-ai.com/">
+    <img
+      alt="Zir AI"
+      src="https://zir-ai.com/static/media/logo-light.637c616a.svg"
+      width="400"
+    />
+  </a>
+</p>
+
+The source code in this repository shows a simple Rasa chatbot fallback
+mechanism, which gets relevant information from ZIR Semantic Search.
+
+# Table of contents
+
+- [Rasa Bot & Zir](#rasa-bot--zir)
+- [Setup & Play](#setup--play)
+  - [Set-up Rasa](#set-up-rasa)
+  - [Set-up repo](#set-up-repo)
+  - [Running the demo](#running-the-demo)
+- [Customizing the bot](#customizing-the-bot)
+  - [Data](#data)
+  - [Rasa Custom Action](#rasa-custom-action)
+
+## Rasa Bot & Zir
+
+[Rasa](https://rasa.com/) is the leading conversational AI platform, for
+personalized conversations at scale. Developers provide the questions they
+expect the end customer to ask and the Rasa bot, using AI, predicts and matches
+what the customer intended to ask. With this information at hand, developers can
+easily match what the bot should respond to.
+
+This is all well unless when the customer asks a valid question the developer
+did not expect, for such a scenario Rasa provides a
+[fallback mechanism](https://rasa.com/docs/rasa/fallback-handoff#fallbacks).
+
+> Although Rasa will generalize to unseen messages, some messages might receive
+> a low classification confidence. Using Fallbacks will help ensure that these
+> low confidence messages are handled gracefully, giving your assistant the
+> option to either respond with a default message or attempt to disambiguate the
+> user input.
+
+Using the mechanism the bot can ask a fallback action and search through the
+reviews uploaded in [Zir-AI](https://zir-ai.com) with the help of
+[custom actions](https://rasa.com/docs/rasa/custom-actions). The relevant
+information is shown if available otherwise if the confidence of the question to
+reviews available is low, we fallback to a generic statement.
+
+The results sum up to be this:
+
+![zir-serach](./web/rasa_zir.gif)
+
+## Setup & Play
+
+You'll need to have Rasa installed and this repository. You might want to create
+a [virtual environment](https://docs.python.org/3/library/venv.html) to isolate
+the dependencies rasa requires.
+
+### Set-up Rasa
+
+Follow the
+[rasa installation instructions](https://rasa.com/docs/rasa/installation) or
+copy-paste the following commands in terminal
+
+```bash
+pip3 install -U pip
+pip3 install rasa
+rasa --version
+```
+
+This should print the rasa version similar to follow
+
+```bash
+Rasa Version      :         2.7.0
+Minimum Compatible Version: 2.6.0
+Rasa SDK Version  :         2.7.0
+Rasa X Version    :         None
+Python Version    :         3.8.5
+Operating System  :         Linux
+Python Path       :         /bin/python3
+```
+
+> Windows & WSL works too
+
+### Set-up repo
+
+To install dependencies for this repository to work install python dependencies
+from requirements.txt
+
+```bash
+pip install -r requirements.txt
+python3 -m spacy download en_core_web_md
+```
+
+This will install the spacy model `en_core_web_md` this bot is configured with.
+Now you'll need to train the rasa bot
+
+```bash
+rasa train
+```
+
+### Running the demo
+
+While running the model, you are required to run the rasa action server along
+with the rasa bot. So in one terminal run
+
+```bash
+rasa run actions
+```
+
+In another terminal, you can either play with the bot in your `shell` or on the
+browser.
+
+To talk to the bot in shell run
+
+```bash
+rasa shell
+```
+
+Or to talk to the bot in the browser run
+
+```bash
+rasa run --credentials ./credentials.yml  --enable-api --auth-token XYZ123 --model ./models --endpoints ./endpoints.yml --cors "*"
+```
+
+Now, open the `index.html` in your browser to talk to the bot.
+
+## Customizing the bot
+
+Use the following information to customize the bot and the source of data.
+
+### Data
+
+Data for the bot is sourced from [OpinRank](https://github.com/kavgan/OpinRank/)
+and reviews for Hotel Jumeirah are specifically used for this bot. Reviews are
+parsed from https://github.com/amin3141/zir-souffle. Use `hotels.py` to generate
+the output `json` and the `reviews.db`. Copy the generated `reviews.db` file to
+this repository.
+
+Create a corpus on [Zir AI](https://zir-ai.com/console/corpora) and upload all
+the JSON files. Create an API key for access and note down the following
+information
+
+- API key
+- corpus id
+- customer id
+
+### Rasa Custom Action
+
+Use the above-collected information and replace the information in
+`actions/actions.py` and you're good to go.
+
+```python
+customer_id = "1835841754"
+corpus_id = 1
+header = {
+    "customer-id": customer_id,
+    "x-api-key": "zqt_WvU_2atFHgqYxxT2sQswwIUgogI8K3QeWs0oqA"
+}
+```

actions/actions.py

@@ -0,0 +1,168 @@
+"""
+This files contains your custom actions which can be used 
+to run custom Python code.
+"""
+from typing import Any, Text, Dict, List
+import json
+import requests
+import sqlite3
+import textwrap
+
+
+from rasa_sdk import Action, Tracker
+from rasa_sdk.events import UserUtteranceReverted
+from rasa_sdk.executor import CollectingDispatcher
+
+
+class ActionDefaultFallback(Action):
+    """Executes the fallback action and goes back to the previous state
+    of the dialogue"""
+
+    customer_id = "1526022105"
+    corpus_id = 40
+    header = {
+        "customer-id": customer_id,
+        "x-api-key": "zqt_WvU_2atFHgqYxxT2sQswwIUgogI8K3QeWs0oqA"
+    }
+    con = None
+
+    def name(self) -> Text:
+        con = sqlite3.connect("./reviews.db")
+        self.cur = con.cursor()
+        return "zir_action_fallback"
+
+    async def run(
+        self,
+        dispatcher: CollectingDispatcher,
+        tracker: Tracker,
+        domain: Dict[Text, Any],
+    ) -> List[Dict[Text, Any]]:
+
+        number_results = 3
+        data_dict = {
+            "query": [
+                {
+                    "query": tracker.latest_message["text"],
+                    "num_results": number_results,
+                    "corpus_key": [
+                        {
+                            "customer_id": self.customer_id,
+                            "corpus_id": self.corpus_id
+                        }
+                    ]
+                }
+            ]
+        }
+
+        payload = json.dumps(data_dict)
+        response = requests.post(f"https://h.serving.zir-ai.io:443/v1/query",
+                                 data=payload,
+                                 verify=True,
+                                 headers=self.header,
+                                 timeout=10)
+        parsed = json.loads(response.text)
+        first_resp = parsed["responseSet"][0]["response"][0]
+        last_resp = parsed["responseSet"][0]["response"][-1]
+        textQuery = []
+        print(last_resp["score"])
+        print(first_resp["score"])
+        if last_resp["score"] < 0.3 or first_resp["score"] < 0.3:
+            textQuery.append(
+                "I'm sorry, I don't have any information about that.")
+        else:
+            textQuery = print_responses(parsed["responseSet"][0], self.cur)
+            textQuery.insert(0, "\n")
+            textQuery.insert(
+                0, "Here's what other customers had to say about that:")
+        textQuery = "\n".join(textQuery)
+        dispatcher.utter_message(text=textQuery)
+
+        # Revert user message which led to fallback.
+        return [UserUtteranceReverted()]
+
+
+def print_responses(response_set, sqlite_cursor):
+    """Print responses to the console."""
+    text_list = []
+    for result in response_set["response"]:
+        doc = response_set["document"][result["documentIndex"]]
+        query = f"""
+            SELECT title, date, hotel, review FROM reviews
+                WHERE doc_id="{doc["id"]}"
+        """
+        for row in sqlite_cursor.execute(query):
+            title, date, hotel, fulltext = row
+            text_list.append("**" + title + "**")
+            if is_title(result):
+                text_list.append(f"{head(fulltext)}")
+            else:
+                t = result["text"]
+                text_list.append(f"{highlight(fulltext, t)}")
+            text_list.append(
+                "*" + f"{" > \"Jumeirah Hotels & Resorts\" reviewed on {date}" + " *")
+            text_list.append("\n")
+            break
+    return text_list
+
+
+def highlight(fulltext, snippet):
+    """Return a result snippet with context, suitable for terminal display."""
+    if snippet in fulltext:
+        start = fulltext.index(snippet)
+        end = start + len(snippet)
+
+        lines = textwrap.wrap(fulltext)
+        start_line = 0
+        end_line = len(lines)
+        pos = 0
+
+        # Figure out which lines to display, and insert ANSI
+        # code to highlight the actual snippet.
+        for x, line in enumerate(lines):
+            next_pos = pos + len(line)
+
+            color_start = pos <= start < next_pos
+            color_end = pos <= end < next_pos
+
+            if color_start and color_end:
+                start_line = end_line = x
+                ips = start - pos - x   # insertion point
+                ipe = end - pos - x     # insertion point
+                lines[x] = line[:ips] + "**" + line[ips:ipe] + \
+                    "**" + line[ipe:]
+            elif color_start:
+                start_line = x
+                ip = start - pos - x    # insertion point
+                lines[x] = line[:ip] + "**" + line[ip:]
+            elif color_end:
+                end_line = x
+                ip = end - pos - x    # insertion point
+                lines[x] = line[:ip] + "**" + line[ip:]
+
+            pos = next_pos
+
+        # Widen the line selection to include a bit of context.
+        if start_line > 0:
+            start_line -= 1
+        end_line += 2
+        return prettify('\n'.join(lines[start_line:end_line]))
+    return prettify(snippet)
+
+
+def head(fulltext):
+    """Returns the first three lines of the review."""
+    lines = textwrap.wrap(fulltext)
+    return prettify('\n'.join(lines[0:3]) + '...')
+
+
+def is_title(result):
+    """Returns true if the result is a title match."""
+    for metadatum in result["metadata"]:
+        if metadatum["name"] == "is_title":
+            return metadatum["value"] == "true"
+    return False
+
+
+def prettify(text):
+    """Clean up the text to make it more suitable for display."""
+    return text.replace("&amp;", "&").replace("&quot;", "\"")

config.yml

@@ -0,0 +1,27 @@
+# Configuration for Rasa NLU.
+# https://rasa.com/docs/rasa/nlu/components/
+language: en
+
+pipeline:
+  - name: SpacyNLP
+  - name: SpacyTokenizer
+  - name: SpacyFeaturizer
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: "char_wb"
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+    epochs: 100
+  - name: EntitySynonymMapper
+  - name: ResponseSelector
+    epochs: 100
+  - name: FallbackClassifier
+    threshold: 0.6
+    ambiguity_threshold: 0.1
+
+# Configuration for Rasa Core.
+# https://rasa.com/docs/rasa/core/policies/
+policies:

credentials.yml

@@ -0,0 +1,6 @@
+# This file contains the credentials for the voice & chat platforms
+# which your bot is using.
+# https://rasa.com/docs/rasa/messaging-and-voice-channels
+
+rasa:
+  url: "http://localhost:5002/api"

data/nlu.yml

@@ -0,0 +1,101 @@
+# NLU training data for the bot. 
+# This file stores the expected queries from the user.
+# Whenever a user utters a message, it is always matched to an intent.
+# This match comes with a confidence number that can be used to identify 
+# how much related the intent is to the uttered message.
+# Minumum number of examples are 2 for each intent. 
+# When writing a new intent, update domain.yml as well.
+version: "2.0"
+
+nlu:
+- intent: greet
+  examples: |
+    - hey
+    - hello
+    - hi
+    - hello there
+    - good morning
+    - good evening
+    - moin
+    - hey there
+    - let's go
+    - hey dude
+    - goodmorning
+    - goodevening
+    - good afternoon
+
+- intent: goodbye
+  examples: |
+    - good afternoon
+    - cu
+    - good by
+    - cee you later
+    - good night
+    - bye
+    - goodbye
+    - have a nice day
+    - see you around
+    - bye bye
+    - see you later
+
+- intent: affirm
+  examples: |
+    - yes
+    - y
+    - indeed
+    - of course
+    - that sounds good
+    - correct
+
+- intent: deny
+  examples: |
+    - no
+    - n
+    - never
+    - I don't think so
+    - don't like that
+    - no way
+    - not really
+
+- intent: ask_location
+  examples: |
+    - Where is the hotel? 
+    - The hotel is where? 
+    - In which city is the hotel? 
+
+- intent: jumeirah_living
+  examples: |
+    - What is Jumeirah Living?
+    - Is Jumeirah Living a package?
+    - What can yo tell of Jumeirah Living?
+    - Jumeirah Living?
+    - Jumeirah?
+
+- intent: jumeirah_one
+  examples: |
+    - What is Jumeirah One?
+    - Is Jumeirah One a package?
+    - What can yo tell of Jumeirah One?
+    - Jumeirah One?
+    - Jumeirah?
+
+- intent: lost_and_found
+  examples: |
+    - What is your policy on list and found items?
+    - I lost something, can it be returned?
+    - Luggage lost
+    - Lost my phone.
+    - Return a lost items?
+
+- intent: jumeirah
+  examples: |
+    - What is Jumeirah?
+    - Is Jumeirah a hotel or a brand?
+
+- intent: reservations
+  examples: |
+    - How many rooms do you have?
+    - i want to reserve rooms
+    - Rooms available
+    - Book a room
+    - Need a place to stay for the night

data/rules.yml

@@ -0,0 +1,16 @@
+# Rules, along with stories are used as training data for the bot and define 
+# how a conversation should always happen provided an intent. 
+# If in doubt, use stories and do not over-use rules.
+version: "2.0"
+
+rules:
+
+- rule: Say goodbye anytime the user says goodbye
+  steps:
+  - intent: goodbye
+  - action: utter_goodbye
+
+- rule: Ask the user to rephrase whenever they send a message with low NLU confidence
+  steps:
+  - intent: nlu_fallback
+  - action: zir_action_fallback

data/stories.yml

@@ -0,0 +1,41 @@
+# This file presents stories or the conversations likely to happen 
+# with the user. Stories are also training data for the bot to stitch intents
+# with actions & to behave more like a human.
+version: "2.0"
+
+stories:
+
+- story: faq question
+  steps:
+   - intent: greet
+   - action: utter_greet
+   - intent: reservations
+   - action: utter_reservation
+
+- story: faq jum liv
+  steps:
+   - intent: greet
+   - action: utter_greet
+   - intent: jumeirah_living
+   - action: utter_jum_liv
+     
+- story: faq jum one
+  steps:
+   - intent: greet
+   - action: utter_greet
+   - intent: jumeirah_one
+   - action: utter_jum_one
+     
+- story: faq jum
+  steps:
+   - intent: greet
+   - action: utter_greet
+   - intent: jumeirah
+   - action: utter_jum
+     
+- story: faq lost
+  steps:
+   - intent: greet
+   - action: utter_greet
+   - intent: lost_and_found
+   - action: utter_lost_found

domain.yml

@@ -0,0 +1,54 @@
+# This file binds the different aspects of your bot together. 
+# Stories, intents, actions etc. are all declared here to be used.
+# To learn more https://rasa.com/docs/rasa/domain/
+version: "2.0"
+
+intents:
+  - greet
+  - goodbye
+  - affirm
+  - deny
+  - jumeirah_living
+  - jumeirah_one
+  - lost_and_found
+  - jumeirah
+  - reservations
+  - ask_location
+
+responses:
+  utter_please_rephrase:
+  - text: Sorry I didn't get that. Can you rephrase?
+    
+  utter_greet:
+  - text: "Hey! How are you?"
+
+  utter_did_that_help:
+  - text: "Did that help you?"
+
+  utter_happy:
+  - text: "Great, carry on!"
+
+  utter_goodbye:
+  - text: "Bye"
+    
+  utter_reservation:
+  - text: "Please go to 'Check Rates & Book Now' section of the hotel you have a reservation for. You will find and an area called 'Cancellations and Amendments'. For all other amendments and cancellations, email or call our Reservations Office."
+
+  utter_jum_liv:
+  - text: "Jumeirah Living is our luxury brand of residences and serviced apartments. Whether for short, mid-term or extended stays, Jumeirah Living will offer innovative yet personal lifestyle experiences unique to each guest, resident and owner, with a strong emphasis on effortless living."
+
+  utter_jum_one:
+  - text: "Jumeirah One is Jumeirah’s loyalty programme designed to recognise and reward loyal guests of Jumeirah. Guests can enjoy more of life’s pleasures with privileged access to an exclusive global lifestyle."
+    
+  utter_lost_found:
+  - text: "Should any guests lose personal belongings whilst staying at a Jumeirah Hotel or Resort in Dubai, if recovered, the item will be recorded as 'found' by that particular property. Each property keeps detailed records of all 'found' items and will store these for a maximum of three months. After this period, in line with the directives of the General Department of Criminal Investigation in Dubai, all unclaimed items will be handed over to the nearest Police Station. These 'found' items will then be kept at the police warehouse for a further three months after which they will be auctioned, and any funds raised given to charity.Should hotel guests need to locate any item after the initial three month period expires, they are requested to contact the hotel who will assist in coordinating with the police station where the found items are handed over. The police will then advise the necessary procedures to locate any item."
+      
+  utter_jum:
+  - text: "Jumeirah is a world-class international hotel and hospitality company which currently manages hotels in  Abu Dhabi, Bahrain, Dubai, Frankfurt, Kuwait, London, Maldives, Mallorca and Shanghai. Please view our Portfolio for more information."
+    
+session_config:
+  session_expiration_time: 60
+  carry_over_slots_to_new_session: true
+
+actions:
+  - zir_action_fallback

endpoints.yml

@@ -0,0 +1,4 @@
+# This file contains the different endpoints your bot can use.
+
+action_endpoint:
+ url: "http://0.0.0.0:5055/webhook"

requirements.txt

@@ -0,0 +1,3 @@
+sqlite3
+textwrap
+spacy

tests/test_stories.yml

@@ -0,0 +1,26 @@
+#### This file contains tests to evaluate that your bot behaves as expected.
+#### If you want to learn more, please see the docs: https://rasa.com/docs/rasa/testing-your-assistant
+stories:
+
+- story: A basic story test
+  steps:
+  - user: |
+      hello
+    intent: greet
+  - action: utter_greet
+  - user: |
+     Book a room for two please.
+    intent: reservations
+  - action: utter_reservation
+    
+
+- story: Fallback test
+  steps:
+  - user: |
+      hello
+    intent: greet
+  - action: utter_greet
+  - user: |
+     Can we play golf?
+    intent: nlu_fallback
+  - action: zir_action_fallback

web/index.html

@@ -0,0 +1,21 @@
+<head>
+    <link rel="stylesheet"
+        href="https://npm-scalableminds.s3.eu-central-1.amazonaws.com/@scalableminds/chatroom@master/dist/Chatroom.css" />
+</head>
+
+<body>
+    <div class="chat-container"></div>
+
+    <script
+        src="https://npm-scalableminds.s3.eu-central-1.amazonaws.com/@scalableminds/chatroom@master/dist/Chatroom.js" />
+    </script>
+    <script type="text/javascript">
+        var chatroom = new window.Chatroom({
+            host: "http://localhost:5005",
+            title: "Kendra, powered by Rasa and ZIR AI",
+            container: document.querySelector(".chat-container"),
+            welcomeMessage: "Hello, my name is Kendra, and I'll be happy to assist with your questions about Burj Al Arab Jumeirah. How may I help you"
+        });
+        chatroom.openChat();
+    </script>
+</body>