Update README.md

README.md

@@ -1,3 +1,148 @@
 ---
 license: mit
+datasets:
+- teknium/openhermes
 ---
+
+
+# Conversational Language Model Interface using FASTTEXT 
+
+This project provides a Command Line Interface (CLI) for interacting with a FastText language model, enabling users to generate text sequences based on their input. The script allows customization of parameters such as temperature, input text, top-k predictions, and model file path.
+
+## Installation
+
+Before running the script, ensure you have Python installed on your system. Additionally, you'll need to install the FastText library:
+
+## Colab 
+
+[Google Colab Notebook](https://colab.research.google.com/drive/1jX1NShX7MzJnuL2whHNOA39Xu-meQ1ap?usp=sharing)
+
+
+```bash
+pip install fasttext
+```
+
+## Usage
+
+To use the script, you should first obtain or train a FastText model. Place the model file (usually with a `.bin` extension) in a known directory.
+
+The script can be executed with various command-line arguments to specify the behavior:
+```python
+import argparse
+import fasttext
+import numpy as np
+
+def apply_repetition_penalty(labels, probabilities, used_labels, penalty_scale=1.9):
+    """
+    Applies a repetition penalty to reduce the probability of already used labels.
+
+    :param labels: List of possible labels.
+    :param probabilities: Corresponding list of probabilities.
+    :param used_labels: Set of labels that have already been used.
+    :param penalty_scale: Scale of the penalty to be applied.
+    :return: Adjusted probabilities.
+    """
+    adjusted_probabilities = probabilities.copy()
+    for i, label in enumerate(labels):
+        if label in used_labels:
+            adjusted_probabilities[i] /= penalty_scale
+    # Normalize the probabilities to sum to 1 again
+    adjusted_probabilities /= adjusted_probabilities.sum()
+    return adjusted_probabilities
+
+def predict_sequence(model, text, sequence_length=20, temperature=.5, penalty_scale=1.9):
+    """
+    Generates a sequence of labels using the FastText model with repetition penalty.
+
+    :param model: Loaded FastText model.
+    :param text: Initial text to start the prediction from.
+    :param sequence_length: Desired length of the sequence.
+    :param temperature: Temperature for sampling.
+    :param penalty_scale: Scale of repetition penalty.
+    :return: List of predicted labels.
+    """
+    used_labels = set()
+    sequence = []
+
+    for _ in range(sequence_length):
+        # Predict the top k most probable labels
+        labels, probabilities = model.predict(text, k=40)
+        labels = [label.replace('__label__', '') for label in labels]
+        probabilities = np.array(probabilities)
+
+        # Adjust the probabilities with repetition penalty
+        probabilities = apply_repetition_penalty(labels, probabilities, used_labels, penalty_scale)
+
+        # Sampling according to the adjusted probabilities
+        label_index = np.random.choice(range(len(labels)), p=probabilities)
+        chosen_label = labels[label_index]
+
+        # Add the chosen label to the sequence and to the set of used labels
+        sequence.append(chosen_label)
+        used_labels.add(chosen_label)
+
+        # Update the text with the chosen label for the next prediction
+        text += ' ' + chosen_label
+
+    return sequence
+
+def generate_response(model, input_text, sequence_length=512, temperature=.5, penalty_scale=1.9):
+    generated_sequence = predict_sequence(model, input_text, sequence_length, temperature, penalty_scale)
+    return ' '.join(generated_sequence)
+
+def main():
+    parser = argparse.ArgumentParser(description="Run the language model with specified parameters.")
+    parser.add_argument('-t', '--temperature', type=float, default=0.5, help='Temperature for sampling.')
+    parser.add_argument('-f', '--file', type=str, help='File containing input text.')
+    parser.add_argument('-p', '--text', type=str, help='Direct input text.')
+    parser.add_argument('-n', '--length', type=int, default=50, help='length predictions to consider.')
+    parser.add_argument('-m', '--model', type=str, required=True, help='Address of the FastText model file.')
+
+    args = parser.parse_args()
+
+    # Load the model
+    model = fasttext.load_model(args.model)
+
+    input_text = ''
+    if args.file:
+        with open(args.file, 'r') as file:
+            input_text = file.read()
+    elif args.text:
+        input_text = args.text
+    else:
+        print("No input text provided. Please use -f to specify a file or -p for direct text input.")
+        return
+
+    # Generate and print the response
+    response = generate_response(model, input_text + " [RESPONSE]", sequence_length=args.length, temperature=args.temperature)
+    print("\nResponse:")
+    print(response)
+
+if __name__ == "__main__":
+    main()
+
+
+```
+
+```bash
+python conversation_app.py -t TEMPERATURE -f FILE -p TEXT -k TOPK -m MODEL_PATH
+```
+
+- `-t TEMPERATURE` or `--temperature TEMPERATURE`: Sets the temperature for predictions. A higher temperature results in more diverse results. Default is 0.5.
+- `-f FILE` or `--file FILE`: Specifies a path to a file containing input text. The script will read this file and use its contents as input.
+- `-p TEXT` or `--text TEXT`: Directly provide the input text as a string.
+- `-n LENGTH` or `--length TOPK`: Determines the number of top predictions to consider for the model's output. Default is 50.
+- `-m MODEL_PATH` or `--model MODEL_PATH`: The path to the FastText model file (required).
+
+### Example
+
+```bash
+python conversation_app.py -t 0.7 -p "What is the future of AI?" -n 40 -m /path/to/model.bin
+```
+
+This command sets the temperature to 0.7, uses the provided question as input, considers the top 40 predictions, and specifies the model file path.
+
+## Note
+
+- The script's output depends on the quality and training of the FastText model used.
+- Ensure the specified model file path and input file path (if used) are correct.