v1

.gitattributes

@@ -32,3 +32,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+evals/evals/registry/data/formal_logic/formal_logic_expressions.jsonl filter=lfs diff=lfs merge=lfs -text
+evals/evals/registry/data/ukraine_eit/samples.jsonl filter=lfs diff=lfs merge=lfs -text

evals/.gitattributes

@@ -0,0 +1 @@
+evals/registry/data/**/*.jsonl filter=lfs diff=lfs merge=lfs -text

evals/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,87 @@
+# Thank you for contributing an eval! ♥️
+
+🚨 Please make sure your PR follows these guidelines, __failure to follow the guidelines below will result in the PR being closed automatically__. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨
+
+__PLEASE READ THIS__:
+
+In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.
+
+We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.
+
+## Eval details 📑
+### Eval name
+[Insert Eval name here]
+
+### Eval description
+
+[Insert a short description of what your eval does here]
+
+### What makes this a useful eval?
+
+[Insert why this eval is worth including and any additional context]
+
+## Criteria for a good eval ✅
+
+Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).
+
+Your eval should be:
+
+- [ ] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can  create an eval on cases where the model fails to reason about the physical world.
+- [ ] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
+- [ ] Includes good signal around what is the right behavior. This means either a correct answer for `Basic` evals or the `Fact` Model-graded eval, or an exhaustive rubric for evaluating answers for the `Criteria` Model-graded eval.
+- [ ] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)
+
+If there is anything else that makes your eval worth including, please document it below.
+
+### Unique eval value
+
+> Insert what makes your eval high quality that was not mentioned above. (Not required)
+
+## Eval structure 🏗️
+
+Your eval should
+- [ ] Check that your data is in `evals/registry/data/{name}`
+- [ ] Check that your yaml is registered at `evals/registry/evals/{name}.yaml`
+- [ ] Ensure you have the right to use the data you submit via this eval
+
+(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)
+
+## Final checklist 👀
+
+### Submission agreement
+
+By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).
+
+- [ ] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.
+
+### Email address validation
+
+If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.
+
+- [ ] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.
+
+### Limited availability acknowledgement
+
+We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.
+
+- [ ] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.
+
+### Submit eval
+
+- [ ] I have filled out all required fields in the evals PR form
+- [ ] (Ignore if not submitting code) I have run `pip install pre-commit; pre-commit install` and have verified that `black`, `isort`, and `autoflake` are running when I commit and push
+
+Failure to fill out all required fields will result in the PR being closed.
+
+### Eval JSON data 
+
+Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:
+
+<details>
+  <summary>View evals in JSON</summary>
+
+  ### Eval
+  ```jsonl
+  INSERT_EVAL_HERE
+  ```
+</details>

evals/.github/bug_report.yml

@@ -0,0 +1,56 @@
+name: Bug report
+description: Create a report to help us improve
+labels: ["bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report! If you have questions about using the OpenAI Evals library, please open a [Discussion thread](https://github.com/openai/evals/discussions). 
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: Describe the bug
+      description: A clear and concise description of what the bug is, and any additional context.
+      placeholder: Tell us what you see!
+    validations:
+      required: true
+  - type: textarea
+    id: repro-steps
+    attributes:
+      label: To Reproduce
+      description: Steps to reproduce the behavior.
+      placeholder: |
+        1. Fetch a '...'
+        2. Update the '....'
+        3. See error
+    validations:
+      required: true
+  - type: textarea
+    id: code-snippets
+    attributes:
+      label: Code snippets
+      description: If applicable, add code snippets to help explain your problem.
+      render: Python
+    validations:
+      required: false
+  - type: input
+    id: os
+    attributes:
+      label: OS
+      placeholder: macOS
+    validations:
+      required: true
+  - type: input
+    id: language-version
+    attributes:
+      label: Python version
+      placeholder: Python v3.8.0
+    validations:
+      required: true
+  - type: input
+    id: lib-version
+    attributes:
+      label: Library version
+      placeholder: openai-evals v0.1.1
+    validations:
+      required: true

evals/.github/config.yml

@@ -0,0 +1,7 @@
+blank_issues_enabled: false
+contact_links:
+  - name: OpenAI support
+    url: https://help.openai.com/
+    about: |
+      Please only file issues here that you believe represent actual bugs or feature requests for the OpenAI Evals library.
+      If you're having general trouble with the OpenAI API, ChatGPT, etc, please visit our help center to get support.

evals/.github/feature_request.yml

@@ -0,0 +1,20 @@
+name: Feature request
+description: Suggest an idea for this library
+labels: ["feature-request"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this feature request! Please note, we are not able to accommodate all feature requests given limited bandwidth but we appreciate you taking the time to share with us how to improve the OpenAI Evals library.
+  - type: textarea
+    id: feature
+    attributes:
+      label: Describe the feature or improvement you're requesting
+      description: A clear and concise description of what you want to happen.
+    validations:
+      required: true
+  - type: textarea
+    id: context
+    attributes:
+      label: Additional context
+      description: Add any other context about the feature request here.

evals/.github/workflows/parse_yaml.py

@@ -0,0 +1,12 @@
+import sys
+import yaml
+
+def get_first_key(file_path):
+    with open(file_path, 'r') as yaml_file:
+        content = yaml.safe_load(yaml_file)
+        first_key = next(iter(content))
+        return first_key
+
+if __name__ == "__main__":
+    yaml_file_path = sys.argv[1]
+    print(get_first_key(yaml_file_path))

evals/.github/workflows/test_eval.yaml

@@ -0,0 +1,55 @@
+name: Run new evals
+
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  check_files:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+      with:
+        fetch-depth: 0
+        lfs: true
+    
+    - name: Install Git LFS
+      run: |
+        sudo apt-get install git-lfs
+        git lfs install
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.9
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install pyyaml
+        pip install -e .
+
+    - name: Get list of new YAML files in evals/registry/evals
+      id: get_files
+      run: |
+        # Use environment files to store the output
+        git diff --name-only --diff-filter=A ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep '^evals/registry/evals/.*\.yaml$' | xargs > new_files
+        echo "new_files=$(cat new_files)" >> $GITHUB_ENV
+
+    - name: Run oaieval command for each new YAML file
+      run: |
+        files="${{ env.new_files }}"
+        if [ -n "$files" ]; then
+          for file in $files; do
+            echo "Processing $file"
+            first_key=$(python .github/workflows/parse_yaml.py $file)
+            echo "Eval Name: $first_key"
+            oaieval dummy-chat $first_key --max_samples 10
+            oaieval dummy-completion $first_key --max_samples 10
+          done
+        else
+          echo "No new YAML files found in evals/registry/evals"
+        fi

evals/.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+evals.egg-info/
+.vscode/

evals/.pre-commit-config.yaml

@@ -0,0 +1,29 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 22.8.0
+    hooks:
+      - id: black
+        args: [--line-length=100, --exclude=""]
+
+  # this is not technically always safe but usually is
+  # use comments `# isort: off` and `# isort: on` to disable/re-enable isort
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+        args: [--line-length=100, --profile=black]
+
+  # this is slightly dangerous because python imports have side effects
+  # and this tool removes unused imports, which may be providing
+  # necessary side effects for the code to run
+  - repo: https://github.com/PyCQA/autoflake
+    rev: v1.6.1
+    hooks:
+      - id: autoflake
+        args:
+          - "--in-place"
+          - "--expand-star-imports"
+          - "--remove-duplicate-keys"
+          - "--remove-unused-variables"
+          - "--remove-all-unused-imports"
+        exclude: "evals/__init__.py"

evals/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 OpenAI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evals/MANIFEST.in

@@ -0,0 +1,3 @@
+recursive-include evals *.py
+recursive-include evals *.yaml
+recursive-include evals *.sql

evals/Makefile

@@ -0,0 +1,2 @@
+mypy:
+	mypy --config-file=mypy.ini --no-site-packages .

evals/README.md

@@ -0,0 +1,89 @@
+This is a fork of the evals repo from OpenAI that allows to evaluate models created outside of OpenAI using the same benchmarks. This provides an opportunity for apple-to-apple comparisons between AGI models of various origins, as long as their input and output specs are aligned.
+
+# Evals
+
+Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.
+
+You can use Evals to create and run evaluations that:
+- use datasets to generate prompts,
+- measure the quality of completions provided by an OpenAI model, and
+- compare performance across different datasets and models.
+
+With Evals, we aim to make it as simple as possible to build an eval while writing as little code as possible. To get started, we recommend that you follow these steps **in order**:
+1. Read through this doc and follow the [setup instructions below](README.md#Setup).
+2. Learn how to run existing evals: [run-evals.md](docs/run-evals.md).
+3. Familiarize yourself with the existing eval templates: [eval-templates.md](docs/eval-templates.md).
+4. Walk through the process for building an eval: [build-eval.md](docs/build-eval.md)
+5. See an example of implementing custom eval logic: [custom-eval.md](docs/custom-eval.md).
+
+If you think you have an interesting eval, please open a PR with your contribution. OpenAI staff actively review these evals when considering improvements to upcoming models.
+
+____________________
+🚨 For a limited time, we will be granting GPT-4 access to those who contribute high quality evals. Please follow the instructions mentioned above and note that spam or low quality submissions will be ignored❗️
+
+Access will be granted to the email address associated with an accepted Eval. Due to high volume, we are unable to grant access to any email other than the one used for the pull request.
+____________________
+
+## Setup
+
+To run evals, you will need to set up and specify your OpenAI API key. You can generate one at <https://platform.openai.com/account/api-keys>. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. **Please be aware of the [costs](https://openai.com/pricing) associated with using the API when running evals.**
+
+**Minimal Required Version: Python 3.9**
+
+### Downloading evals
+
+Our Evals registry is stored using [Git-LFS](https://git-lfs.com/). Once you have downloaded and installed LFS, you can fetch the evals with:
+```sh
+git lfs fetch --all
+git lfs pull
+```
+
+You may just want to fetch data for a select eval. You can achieve this via:
+```sh
+git lfs fetch --include=evals/registry/data/${your eval}
+git lfs pull
+```
+
+### Making evals
+
+If you are going to be creating evals, we suggest cloning this repo directly from GitHub and installing the requirements using the following command:
+
+```sh
+pip install -e .
+```
+
+Using `-e`, changes you make to your eval will be reflected immediately without having to reinstall.
+
+### Running evals
+
+If you don't want to contribute new evals, but simply want to run them locally, you can install the evals package via pip:
+
+```sh
+pip install evals
+```
+
+We provide the option for you to log your eval results to a Snowflake database, if you have one or wish to set one up. For this option, you will further have to specify the `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_DATABASE`, `SNOWFLAKE_USERNAME`, and `SNOWFLAKE_PASSWORD` environment variables.
+
+## FAQ
+
+Do you have any examples of how to build an eval from start to finish?
+
+- Yes! These are in the `examples` folder. We recommend that you also read through [build-eval.md](docs/build-eval.md) in order to gain a deeper understanding of what is happening in these examples.
+
+Do you have any examples of evals implemented in multiple different ways?
+
+- Yes! In particular, see `evals/registry/evals/coqa.yaml`. We have implemented small subsets of the [CoQA](https://stanfordnlp.github.io/coqa/) dataset for various eval templates to help illustrate the differences.
+
+When I run an eval, it sometimes hangs at the very end (after the final report). What's going on?
+
+- This is a known issue, but you should be able to interrupt it safely and the eval should finish immediately after.
+
+There's a lot of code, and I just want to spin up a quick eval. Help? OR,
+
+I am a world-class prompt engineer. I choose not to code. How can I contribute my wisdom?
+
+- If you follow an existing [eval template](docs/eval-templates.md) to build a basic or model-graded eval, you don't need to write any evaluation code at all! Just provide your data in JSON format and specify your eval parameters in YAML. [build-eval.md](docs/build-eval.md) walks you through these steps, and you can supplement these instructions with the Jupyter notebooks in the `examples` folder to help you get started quickly. Keep in mind, though, that a good eval will inevitably require careful thought and rigorous experimentation!
+
+## Disclaimer
+
+By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies: https://platform.openai.com/docs/usage-policies.

evals/SECURITY.md

@@ -0,0 +1,4 @@
+# Security Policy
+For a more in-depth look at our security policy, please check out our [Coordinated Vulnerability Disclosure Policy](https://openai.com/security/disclosure/#:~:text=Disclosure%20Policy,-Security%20is%20essential&text=OpenAI%27s%20coordinated%20vulnerability%20disclosure%20policy,expect%20from%20us%20in%20return.).
+
+Our PGP key can located [at this address.](https://cdn.openai.com/security.txt)

evals/docs/build-eval.md

@@ -0,0 +1,85 @@
+# Building an eval
+
+This document walks through the end-to-end process for building an eval, which is a dataset and a choice of eval class. The `examples` folder contains Jupyter notebooks that follow the steps below to build several academic evals, thus helping to illustrate the overall process.
+
+The steps in this process are building your dataset, registering a new eval with your dataset, and running your eval. Crucially, we assume that you are using an [existing eval template](eval-templates.md) out of the box (if that's not the case, see [this example of building a custom eval](custom-eval.md)). If you are interested in contributing your eval publically, we also include some criteria at the bottom for what we think makes an interesting eval.
+
+We are looking for evals in the following categories:
+
+- Over-refusals
+- Safety
+- System message steerability
+- In-the-wild hallucinations
+- Math / logical / physical reasoning
+- Real-world use case (please describe in your PR how this capability would be used in a product)
+- Other foundational capability
+
+If you have an eval that falls outside this category but still is a diverse example, please contribute it!
+
+## Formatting your data
+
+Once you have an eval in mind that you wish to implement, you will need to convert your samples into the right JSON lines (JSONL) format. A JSONL file is just a JSON file with a unique JSON object per line.
+
+We include some examples of JSONL eval files in [registry/data/README.md](../evals/registry/data/README.md)
+
+Each JSON object will represent one data point in your eval. The keys you need in the JSON object depend on the eval template. All templates expect an `"input"` key which is the prompt, ideally specified in [chat format](https://platform.openai.com/docs/guides/chat/introduction) (though strings are also supported). We recommend chat format even if you are evaluating non chat models. If you are evaluating both chat and non chat models, we handle the conversion between chat formatted prompts and raw string prompts (see the conversion logic [here](../evals/prompt/base.py)).
+
+For the basic evals `Match`, `Includes`, and `FuzzyMatch`, the other required key is `"ideal"`, which is a string (or a list of strings) specifying the correct reference answer(s). For model-graded evals, the required keys vary based on the eval but is determined by the `{key}`s in the evaluation `prompt` that are not covered by the (optional) `args`.
+
+We have implemented small subsets of the [CoQA](https://stanfordnlp.github.io/coqa/) dataset for various eval templates to illustrate how the data should be formatted. See [`coqa/match.jsonl`](../evals/registry/data/coqa/match.jsonl) for an example of data that is suitable for the `Match` basic eval template and [`coqa/samples.jsonl`](../evals/registry/data/coqa/samples.jsonl) for data that is suitable for `fact` and `closedqa` model-graded evals. Note that even though these two model-graded evals expect different keys, we can include the superset of keys in our data in order to support both evals.
+
+If the dataset file is on your local machine, put the `jsonl` file in `evals/registry/data/<eval_name>/samples.jsonl`. If it is in Cloud Object Storage, we support path-style URLs for the major clouds (for your personal use only, we will not accept PRs with cloud URLs).
+
+## Registering the eval
+
+Register the eval by adding a file to `evals/registry/evals/<eval_name>.yaml` using the elsuite registry format. For example, for a `Match` eval, it would be:
+```
+<eval_name>:
+  id: <eval_name>.dev.v0
+  metrics: [accuracy]
+
+<eval_name>.dev.v0:
+  class: evals.elsuite.basic.match:Match
+  args:
+    samples_jsonl: <eval_name>/samples.jsonl
+```
+
+Upon running the eval, the data will be searched for in `evals/registry/data`, e.g. if `test_match/samples.jsonl` is the provided filepath the data is expected to be in `evals/registry/data/test_match/samples.jsonl`.
+
+The naming convention for evals is in the form `<eval_name>.<split>.<version>`.
+- `<eval_name>` is the eval name, used to group evals whose scores are comparable.
+- `<split>` is the data split, used to further group evals that are under the same `<base_eval>`. E.g., "val", "test", or "dev" for testing.
+- `<version>` is the version of the eval, which can be any descriptive text you'd like to use (though it's best if it does not contain ".").
+
+In general, running the same eval name against the same model should always give similar results so that others can reproduce it. Therefore, when you change your eval, you should bump the version.
+
+## Running the eval
+
+You can now run your eval on your data from the CLI with your choice of model:
+```
+oaieval gpt-3.5-turbo <eval_name>
+```
+Congratulations, you have built your eval! Keep iterating on it until you are confident in the results.
+
+## For model-graded evals: a step-by-step workflow
+
+We expect that the existing model-graded evals such as `fact`, `closedqa`, and `battle` will fit many use cases. However, other use cases may benefit from more customization, e.g., a different evaluation prompt. For these, there will be a bit more work involved, but generally still no coding required!
+
+1. If you can't use an existing model-graded eval, create a new YAML or create a new entry to an existing YAML in `evals/registry/modelgraded` to specify the [parameters](eval-templates.md#parameters-for-model-graded-evals) of your eval. See [`humor.yaml`](../evals/registry/modelgraded/humor.yaml) for an example.
+    - Note that, even if you are creating a new YAML, you may find it easiest to copy an existing YAML as a starting point. For example, model-graded evals which check a model completion against a rubric can copy `closedqa.yaml` and just edit the `args`.
+2. Next, you will create your dataset and register your eval, as described above. See [`joke_fruits_labeled.jsonl`](../evals/registry/data/test_metaeval/joke_fruits_labeled.jsonl) and [`joke-fruits`](../evals/registry/evals/test-modelgraded.yaml), for example.
+    - Note that it is recommended to specify `eval_type` at this step, when you register your eval, rather than step 1.
+3. Run your eval, e.g., `oaleval gpt-3.5-turbo joke-fruits`.
+4. (Recommended) Add a meta-eval for the model-graded eval! Each model-graded eval comes with a few knobs to tune, mainly `prompt` but also `eval_type`. In order to make sure the eval is of high quality, we recommend each model-graded eval contribution come with "choice labels", which are basically human-provided labels for which evaluation choice the model should have made. As an example (pretending that these jokes are actually funny), see the `"choice"` keys in [`joke_fruits_labeled.jsonl`](../evals/registry/data/test_metaeval/joke_fruits_labeled.jsonl), which are not used by the `joke-fruits` eval but are used by the [`joke-fruits-meta`](../evals/registry/evals/test-modelgraded.yaml) meta-eval right below it . After running the meta-eval, e.g., `oaieval gpt-3.5-turbo joke-fruits-meta`, the report will output `metascore/` accuracies, which should be close to "1.0" for a good model-graded eval.
+
+## Criteria for contributing an eval
+
+Important: if you are contributing code, make sure to run `pip install pre-commit; pre-commit install` before committing and pushing to ensure that `black`, `isort`, and `autoflake` are run.
+
+We are interested in curating a diverse and interesting set of evals on which to improve our models going forward. Here are some criteria for what we consider a good eval.
+- [ ] The eval should be thematically consistent. We'd like to see a number of prompts all revolving around the same use case, subject domain, failure mode, etc.
+- [ ] The eval should be challenging. If GPT-4 or GPT-3.5-Turbo do well on all of the prompts, this is not as interesting. Of course, the eval should also be possible given the models' limitations and constraints. Oftentimes, a good rule of thumb is whether a human (potentially a subject expert) could do well on the prompts.
+- [ ] The eval should be directionally clear. The data should include good signal around what is the right behavior. This means, for example, high-quality reference answers or an exhaustive rubric for evaluating answers.
+- [ ] The eval should be carefully crafted. Before you submit, you should think through whether you have engineered your prompts for good performance, whether you are using the best eval template, whether you have spot checked your results to ensure accuracy, etc.
+
+Once you are ready to contribute your eval publicly, submit a PR and the OpenAI team will be happy to look it over. Make sure to fill out all parts of the template that is prepopulated into the PR message. Note that submitting a PR does not guarantee that OpenAI will eventually merge it. We will run our own checks and use our best judgment when considering which evals to follow up with.

evals/docs/custom-eval.md

@@ -0,0 +1,148 @@
+# How to add a custom eval
+
+This tutorial will walk you through a simple example of writing and adding a custom eval. The example eval will test the model's ability to do basic arithmetic. We will assume that you have followed the setup instructions in the [README](../README.md) and gone through the other docs for how to run and build evals.
+
+When writing your own evals, the primary files of interest are:
+- `evals/api.py`, which provides common interfaces and utilities used by eval creators to sample from models and process the results,
+- `evals/record.py`, which defines the recorder classes which log eval results in different ways, such as to a local JSON file or to a remote Snowflake database, and
+- `evals/metrics.py`, which defines various common metrics of interest.
+
+These files provide a suite of tools for writing new evals. Once you have gone through this tutorial, you can see a more realistic example of these tools in action with the [machine translation](../evals/elsuite/translate.py) [eval example](../examples/lafand-mt.ipynb), which also implements custom eval logic in lieu of using an existing template.
+
+## Create your datasets
+
+The first step is to create the datasets for your eval. Here, we will create toy train and test sets of just two examples each. The test examples are what we will evaluate the model on, and we'll include the train examples as few-shot examples in the prompt to the model.
+
+We will use the new chat format described [here](https://platform.openai.com/docs/guides/chat/introduction). By default, we encourage all evals to be written using chat formatting if you want to evaluate our new models. Under the hood, we [convert](../evals/prompt/base.py) chat formatted data into raw strings for older non chat models.
+
+To create the toy datasets, in your terminal, type:
+```bash
+echo -e '[{"role": "system", "content": "2+2=", "name": "example_user"}, {"role": "system", "content": "4", "name": "example_assistant"}]\n[{"role": "system", "content": "4*4=", "name": "example_user"}, {"role": "system", "content": "16", "name": "example_assistant"}]' > /tmp/train.jsonl
+echo -e '[{"role": "system", "content": "48+2=", "name": "example_user"}, {"role": "system", "content": "50", "name": "example_assistant"}]\n[{"role": "system", "content": "5*20=", "name": "example_user"}, {"role": "system", "content": "100", "name": "example_assistant"}]' > /tmp/test.jsonl
+```
+
+## Create an eval
+
+The next step is to write a Python class that represents the actual evaluation. This class uses your datasets to create prompts, which are passed to the model to generate completions. Evaluation classes generally will inherit from the `evals.Eval` base class (defined in `evals/eval.py`) and will override two methods: `eval_sample` and `run`.
+
+Let's create a file called `arithmetic.py` under the `evals/elsuite` folder. We'll start by defining the eval class. Its `__init__` method will take in the arguments we need (references to the train and test sets) along with other `kwargs` that will be handled by the base class. We'll also define the `run` method which takes in a `recorder` and returns the final metrics of interest.
+
+```python
+import random
+import textwrap
+
+import evals
+import evals.metrics
+
+class Arithmetic(evals.Eval):
+    def __init__(self, train_jsonl, test_jsonl, train_samples_per_prompt=2, **kwargs):
+        super().__init__(**kwargs)
+        self.train_jsonl = train_jsonl
+        self.test_jsonl = test_jsonl
+        self.train_samples_per_prompt = train_samples_per_prompt
+
+    def run(self, recorder):
+        """
+        Called by the `oaieval` CLI to run the eval. The `eval_all_samples` method calls `eval_sample`.
+        """
+        self.train_samples = evals.get_jsonl(self.train_jsonl)
+        test_samples = evals.get_jsonl(self.test_jsonl)
+        self.eval_all_samples(recorder, test_samples)
+
+        # Record overall metrics
+        return {
+            "accuracy": evals.metrics.get_accuracy(recorder.get_events("match")),
+        }
+```
+
+Generally, most `run` methods will follow the same pattern shown here: loading the data, calling `eval_all_samples`, and aggregating the results (in this case, using the `get_accuracy` function in `evals/metrics.py`). `eval_all_samples` takes in both the `recorder` and the `test_samples` and, under the hood, will call the `eval_sample` method on each sample in `test_samples`. So let's write that `eval_sample` method now:
+
+```python
+    def eval_sample(self, test_sample, rng: random.Random):
+        """
+        Called by the `eval_all_samples` method to evaluate a single sample.
+
+        ARGS
+        ====
+        `test_sample`: a line from the JSONL test file
+        `rng`: should be used for any randomness that is needed during evaluation
+
+        This method does the following:
+        1. Generate a prompt that contains the task statement, a few examples, and the test question.
+        2. Check if the model generates the correct answer.
+        """
+        stuffing = rng.sample(self.train_samples, self.train_samples_per_prompt)
+
+        prompt = [
+            {"role": "system", "content": "Solve the following math problems"},
+        ]
+
+        for i, sample in enumerate(stuffing + [test_sample]):
+            if i < len(stuffing):
+                prompt += [
+                    {"role": "system", "content": sample["problem"], "name": "example_user"},
+                    {"role": "system", "content": sample["answer"], "name": "example_assistant"},
+                ]
+            else:
+                prompt += [{"role": "user", "content": sample["problem"]}]
+
+        evals.check_sampled_text(self.model_spec, prompt, expected=sample["answer"])
+```
+You'll notice that `eval_sample` doesn't take the `recorder` as an argument. This is because `eval_all_samples` sets it to be the default recorder before calling `eval_sample`, and the recording utilities defined in `evals/record.py` use the default recorder. In this example, the `eval_sample` method passes off a lot of the heavy lifting to the `evals.check_sampled_text` utility function, which is defined in `evals/api.py`. This utility function queries the model, defined by `self.model_spec`, with the given `prompt` and checks to see if the result matches the `expected` answer (or one of them, if given a list). It then records these matches (or non matches) using the default recorder.
+
+`eval_sample` methods may vary greatly based on your use case. If you are building custom evals, it is a good idea to be familiar with the functions available to you in `evals/record.py`, `evals/metrics.py`, and especially `evals/api.py`.
+
+## Register your eval
+
+The next step is to register your eval in the registry so that it can be run using the `oaieval` CLI.
+
+Let's create a file called `arithmetic.yaml` under the `evals/registry/evals` folder and add an entry for our eval as follows:
+
+```yaml
+# Define a base eval
+arithmetic:
+  # id specifies the eval that this eval is an alias for
+  # in this case, arithmetic is an alias for arithmetic.dev.match-v1
+  # When you run `oaieval davinci arithmetic`, you are actually running `oaieval davinci arithmetic.dev.match-v1`
+  id: arithmetic.dev.match-v1
+  # The metrics that this eval records
+  # The first metric will be considered to be the primary metric
+  metrics: [accuracy]
+  description: Evaluate arithmetic ability
+# Define the eval
+arithmetic.dev.match-v1:
+  # Specify the class name as a dotted path to the module and class
+  class: evals.elsuite.arithmetic:Arithmetic
+  # Specify the arguments as a dictionary of JSONL URIs
+  # These arguments can be anything that you want to pass to the class constructor
+  args:
+    train_jsonl: /tmp/train.jsonl
+    test_jsonl: /tmp/test.jsonl
+```
+
+The `args` field should match the arguments that your eval class `__init__` method expects.
+
+## Run your eval
+
+The final step is to run your eval and view the results.
+
+```sh
+pip install .  # you can omit this if you used `pip install -e .` to install
+oaieval gpt-3.5-turbo arithmetic
+```
+
+If you run with the `gpt-3.5-turbo` model, you should see an output similar to this (we have cleaned up the output here slightly for readability):
+
+```
+% oaieval gpt-3.5-turbo arithmetic
+... [registry.py:147] Loading registry from .../evals/registry/evals
+... [registry.py:147] Loading registry from .../.evals/evals
+... [oaieval.py:139] Run started: <run_id>
+... [eval.py:32] Evaluating 2 samples
+... [eval.py:138] Running in threaded mode with 1 threads!
+100%|██████████████████████████████████████████| 2/2 [00:00<00:00,  3.35it/s]
+... [record.py:320] Final report: {'accuracy': 1.0}. Logged to /tmp/evallogs/<run_id>_gpt-3.5-turbo_arithmetic.jsonl
+... [oaieval.py:170] Final report:
+... [oaieval.py:172] accuracy: 1.0
+... [record.py:309] Logged 6 rows of events to /tmp/evallogs/<run_id>_gpt-3.5-turbo_arithmetic.jsonl: insert_time=2.038ms
+```

evals/docs/eval-templates.md

@@ -0,0 +1,61 @@
+# Existing templates for evals
+
+In using Evals, we have discovered several "templates" that accommodate many different benchmarks. We have implemented these templates in `evals/elsuite` in order to simplify the development of new evals. We believe that, with these templates, many evals will not require any coding to implement! Instead, you'll pick one of the existing templates and simply specify the dataset and parameters.
+
+## Basic eval templates
+
+In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found the following templates to be useful.
+
+For a model completion `a` and a reference list of correct answers `B`, the following evals implement:
+- [`basic/match.py:Match`](../evals/elsuite/basic/match.py): `any([b.startswith(a) for b in B])`
+- [`basic/includes.py:Includes`](../evals/elsuite/basic/includes.py): `any([(a in b) for b in B])`
+- [`basic/fuzzy_match.py:FuzzyMatch`](../evals/elsuite/basic/fuzzy_match.py): `any([(a in b or b in a) for b in B])`
+
+Which eval template you use will depend on your use case. It is always recommended that you inspect the completions from your model, as this will help you determine how and whether to tweak your prompt (or your reference answers) and pick your eval template. Academic benchmarks oftentimes fit the mold of these basic evals, and we have implemented several end-to-end examples of academic evals as Jupyter notebooks in the `examples` folder.
+
+Sometimes, [custom eval logic](custom-eval.md) will better suit your needs. One example of this is the [machine translation](../evals/elsuite/translate.py) [eval example](../examples/lafand-mt.ipynb), in which there is a unique and clearly defined metric that we wish to use in our eval. You should use your best judgment when deciding between custom eval logic, using a basic eval template, or using model-graded evals as described next.
+
+## The model-graded eval template
+
+In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation. In general, the evaluation model and the model being evaluated don't have to be the same, though we will assume that they are here for ease of explanation.
+
+[`modelgraded/classify.py:ModelBasedClassify`](../evals/elsuite/modelgraded/classify.py) implements the main logic behind our model-graded eval template. In short, we get the model's completion to the original prompt, wrap it in an evaluation prompt, and get the model's completion to the evaluation prompt, which we parse into our metrics of interest. Crucially, the evaluation prompt should prime the model to answer in such a way that is easily parsable, e.g., in multiple choice format or with a simple yes/no. We describe some example model-graded evals below, but first we specify the parameters for this eval template.
+
+### Parameters for model-graded evals
+
+Refer to the [`classify.py:ModelBasedClassify`](../evals/elsuite/modelgraded/classify.py) class to see how these parameters are used in the code.
+
+- `prompt`: The evaluation prompt which should take in the model's completion to the original prompt, potentially along with some other information, and steer the model to provide an evaluation that is easily parsable. Portions denoted by curly braces (i.e., `{key}`) are filled in either from the data `input_outputs` or the additional `args` (see below).
+- `input_outputs`: A mapping specifying which inputs to use to generate which completions. For many evals, there will only be a single input-completion pair, though there can be more, e.g., when comparing two completions against each other.
+- `choice_strings`: The choices that we expect the model completion to contain given the evaluation prompt. For example, `"ABCDE"` or `["Yes", "No", "Unsure"]`. Any other choices returned by the model are parsed into `"__invalid__"`.
+- `choice_scores` (optional): A mapping of each choice to its score, which is logged as a metric. For example, if a response of `"Yes"` (resp. `"No"`) indicates that the model's original completion was good (resp. bad), we may assign this choice a score of 1 (resp. 0).
+- `eval_type` (optional): How we expect the model to format its response to the evaluation prompt. Currently the supported options are:
+  - `"cot_classify"` ("chain-of-thought then classify", i.e., reason then answer) expects that the parsable portion of the response (i.e., the portion containing the choice) will be at the end of the completion. We recommend this as the default as it typically provides most accurate model-graded evaluations.
+  - `"classify_cot"` (answer then reason) expects that the model response will contain the choice first.
+  - `"classify"` expects that the model response will only contain the choice.
+
+  There are two ways to specify `eval_type`. The recommended way is in the `evals/registry/evals` YAML file. If done this way, an instruction will automatically be appended to `prompt` to steer the model towards the expected format (see `ANSWER_PROMPTS` in [the code](../evals/elsuite/modelgraded/classify.py)). Alternatively, you may specify `eval_type` in the `evals/registry/modelgraded` YAML, but you will need to include an appropriate instruction directly in the `prompt`.
+- `args` (optional): If specified, multiple evaluation calls will be made where the evaluation prompt is modified for each call with a different set of arguments.
+- `completion_sample_templates` (optional): If specified, determines how the model's output (or outputs, if `multicomp_n > 1`) will be formatted within the completion.
+
+### Example model-graded evals
+
+To instantiate model-graded evals, create a YAML file in `evals/registry/modelgraded` which specifies values for the arguments described above. We have provided a few examples, which illustrate the process for creating a model-graded eval, but which we also believe are general enough to be useful out of the box for many evals.
+
+[`fact.yaml`](../evals/registry/modelgraded/fact.yaml): a factual consistency eval which, given a completion `a` and reference answer `b`, returns:
+- `"A"` if `a` $\subseteq$ `b`, i.e., the submitted answer is a subset of the expert answer and is fully consistent with it.
+- `"B"` if `a` $\supseteq$ `b`, i.e., the submitted answer is a superset of the expert answer and is fully consistent with it.
+- `"C"` if `a` $=$ `b`, i.e., the submitted answer contains all the same details as the expert answer.
+- `"D"` if `a` $\neq$ `b`, i.e., there is a disagreement between the submitted answer and the expert answer.
+- `"E"` if `a` $\approx$ `b`, i.e., the answers differ, but these differences don't matter from the perspective of factuality.
+
+[`closedqa.yaml`](../evals/registry/modelgraded/closedqa.yaml): a question answering eval which, given a prompt containing a question and the necessary information to answer the question, checks whether the model's answer is:
+- relevant, i.e., extracted from the information provided in the prompt,
+- concise, i.e., did not contain unnecessary details or information, and
+- correct, i.e., uses the extracted information to come to the right conclusion.
+
+Note that this eval is implemented more generally as a "criteria-checking" eval which specifies the evaluation prompt as checking a given criterion and feeding in the above desiderata one by one. We believe that many other evals can be implemented by specifying a "rubric" detailing the criteria of interest and following the same prompt and yes/no choices.
+
+[`battle.yaml`](../evals/registry/modelgraded/battle.yaml): a head-to-head eval which compares two model completions for two potentially different prompts. `choice_scores` is used here to log how often the first completion is judged to be better than the second.
+
+We include additional examples which test more specific model capabilities (such as humor) and are thus less generalizable to other evals. However, these examples still serve to illustrate different ways to write evaluation prompts and set up model-graded evals. See [this section](build-eval.md#for-model-graded-evals-a-step-by-step-workflow) for more detailed steps on building model-graded evals.

evals/docs/run-evals.md

@@ -0,0 +1,37 @@
+# How to run evals
+
+We provide two command line interfaces (CLIs): `oaieval` for running a single eval and `oaievalset` for running a set of evals.
+
+## Running an eval
+
+When using the `oaieval` command, you will need to provide both the model you wish to evaluate as well as the eval to run. E.g.,
+```sh
+oaieval gpt-3.5-turbo test-match
+```
+
+In this example, `gpt-3.5-turbo` is the model to evaluate, and `test-match` is the eval to run. The valid model names are those which you have access to via the API. The valid eval names are specified in the YAML files under `evals/registry/evals`, and their corresponding implementations can be found in `evals/elsuite`.
+
+These CLIs can accept various flags to modify their default behavior. For example:
+- If you wish to log to a Snowflake database (which you have already set up as described in the [README](../README.md)), add `--no-local-run`.
+- By default, logging locally or to Snowflake will write to `tmp/evallogs`, and you can change this by setting a different `--record_path`.
+
+You can run `oaieval --help` to see a full list of CLI options.
+
+## Running an eval set
+
+```sh
+oaievalset gpt-3.5-turbo test
+```
+
+Similarly, `oaievalset` also expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.
+
+By default we run with 10 threads, and each thread times out and restarts after 40 seconds. You can configure this, e.g.,
+
+```sh
+EVALS_THREADS=42 EVALS_THREAD_TIMEOUT=600 oaievalset gpt-3.5-turbo test
+```
+Running with more threads will make the eval faster, though keep in mind the costs and your [rate limits](https://platform.openai.com/docs/guides/rate-limits/overview). Running with a higher thread timeout may be necessary if you expect each sample to take a long time, e.g., the data contain long prompts that elicit long responses from the model.
+
+If you have to stop your run or your run crashes, we've got you covered! `oaievalset` records the evals that finished in `/tmp/oaievalset/{model}.{eval_set}.progress.txt`. You can simply rerun the command to pick up where you left off. If you want to run the eval set starting from the beginning, delete this progress file.
+
+Unfortunately, you can't resume a single eval from the middle. You'll have to restart from the beginning, so try to keep your individual evals quick to run.

evals/evals/__init__.py

@@ -0,0 +1,4 @@
+from .api import check_sampled_text, completion_query, sample_freeform
+from .base import ModelSpec, ModelSpecs
+from .data import get_csv, get_json, get_jsonl, get_jsonls, get_lines, iter_jsonls
+from .eval import Eval

evals/evals/api.py

@@ -0,0 +1,263 @@
+"""
+This file provides common interfaces and utilities used by eval creators to
+sample from models and process the results.
+"""
+
+import logging
+from typing import Callable, Optional, Union
+
+from evals.base import ModelSpec
+from evals.prompt.base import (
+    ChatCompletionPrompt,
+    CompletionPrompt,
+    OpenAICreateChatPrompt,
+    OpenAICreatePrompt,
+    Prompt,
+)
+from evals.record import record_match, record_sampling
+from evals.utils.api_utils import (
+    openai_chat_completion_create_retrying,
+    openai_completion_create_retrying,
+    agi_completion_create_retrying,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def completion_query(
+    model_spec: ModelSpec,
+    prompt: Union[OpenAICreatePrompt, OpenAICreateChatPrompt, Prompt],
+    **kwargs,
+) -> tuple[dict, Union[OpenAICreatePrompt, OpenAICreateChatPrompt], dict]:
+    """
+    Query the API for a completion.
+
+    ARGS
+    ====
+    `model_spec`: `ModelSpec` containing model details to use in the query.
+        This should be the dict returned by `registry.get_model()`.
+        If `model_spec` is not provided, we use the default model that was
+            intialized at the beginning of the run.
+    `prompt`: Either a `Prompt` object or a raw prompt that will get wrapped in
+        the approriate `Prompt` class.
+    `kwargs`: Other arguments passed to the API.
+
+    RETURNS
+    =======
+    The result of the API call.
+    The prompt that was fed into the API call as a str.
+    A dict containing metadata about the query.
+    """
+    if not isinstance(prompt, Prompt):
+        assert (
+            isinstance(prompt, str)
+            or (isinstance(prompt, list) and all(isinstance(token, int) for token in prompt))
+            or (isinstance(prompt, list) and all(isinstance(token, str) for token in prompt))
+            or (isinstance(prompt, list) and all(isinstance(msg, dict) for msg in prompt))
+        ), f"Got type {type(prompt)}, with val {type(prompt[0])} for prompt, expected str or list[int] or list[str] or list[dict[str, str]]"
+
+        if model_spec.is_chat:
+            prompt = ChatCompletionPrompt(
+                raw_prompt=prompt,
+            )
+        else:
+            prompt = CompletionPrompt(
+                raw_prompt=prompt,
+            )
+
+    openai_create_prompt: Union[
+        OpenAICreatePrompt, OpenAICreateChatPrompt
+    ] = prompt.to_openai_create_prompt()
+
+    extra_args = {
+        key: model_spec.extra_options.get(key, kwargs.get(key))
+        for key in set(kwargs) | set(model_spec.extra_options)
+    }
+
+    if model_spec.is_agi:
+        result = agi_completion_create_retrying(
+            model=model_spec.model,
+            api_base=model_spec.api_base,
+            messages=openai_create_prompt,
+            **extra_args,
+        )
+    elif model_spec.is_chat:
+        result = openai_chat_completion_create_retrying(
+            model=model_spec.model,
+            api_base=model_spec.api_base,
+            api_key=model_spec.api_key,
+            messages=openai_create_prompt,
+            **extra_args,
+        )
+    else:
+        result = openai_completion_create_retrying(
+            model=model_spec.model,
+            api_base=model_spec.api_base,
+            api_key=model_spec.api_key,
+            prompt=openai_create_prompt,
+            **extra_args,
+        )
+
+    metadata = {}
+    if result:
+        metadata["completion_id"] = result.get("id", None)
+        metadata["model"] = result.get("model", None)
+
+        if model_spec.is_chat:
+            for choice in result["choices"]:
+                choice["text"] = choice["message"]["content"]
+
+    return result, openai_create_prompt, metadata
+
+
+def check_sampled_text(
+    model_spec: ModelSpec,
+    prompt: Union[OpenAICreatePrompt, OpenAICreateChatPrompt, Prompt],
+    expected: Union[str, list[str], tuple[str]],
+    *,
+    options: Optional[list[str]] = None,
+    separator: Callable[[str], bool] = None,
+) -> Optional[str]:
+    """
+    Generates a completion using the prompt, checks whether the completion is
+        one of the expected completions, and then records the result.
+
+    ARGS
+    ====
+    `model_spec`: See `completion_query`.
+    `prompt`: See `completion_query`.
+    `options`: The list of canonical options, defaults to `expected` if None.
+        The completion will be converted to one of these options.
+    `expected`: The desired completion or the list of desired completions.
+    `separator`: A callable which check the character sampled after the option
+        to see if it is a valid separator.
+
+    RETURNS
+    =======
+    The option that was picked, i.e., matched the completion, or None.
+    """
+    if isinstance(expected, tuple):
+        expected = list(expected)
+    elif not isinstance(expected, list):
+        expected = [expected]
+    if options is None:
+        options = expected
+
+    result, actual_prompt, metadata = completion_query(
+        prompt=prompt,
+        temperature=0.0,
+        model_spec=model_spec,
+    )
+    choice = result["choices"][0]
+
+    sampled = choice["text"].strip() if model_spec.strip_completion else choice["text"]
+
+    picked = None
+    for option in options:
+        if not sampled.startswith(option):
+            continue
+        if (
+            separator is not None
+            and len(sampled) > len(option)
+            and not separator(sampled[len(option)])
+        ):
+            continue
+        picked = option
+        break
+
+    result = {
+        "prompt": actual_prompt,
+        "sampled": sampled,
+        "options": options,
+        "picked": picked,
+    }
+    match = picked in expected
+    result["expected"] = expected
+    result["match"] = match
+    result["metadata"] = metadata
+    record_sampling(**result)
+    record_match(match, expected=expected, picked=picked, sampled=sampled)
+    return picked
+
+
+def sample_freeform(
+    model_spec: ModelSpec,
+    prompt: Union[OpenAICreatePrompt, OpenAICreateChatPrompt, Prompt],
+    *,
+    temperature: float = 1.0,
+    top_p: float = 0.9,
+    max_tokens: int = 512,
+    stop: Optional[str] = None,
+    n_samples: int = None,
+    return_logprobs: bool = False,
+    **kwargs,
+) -> Union[str, list[str], dict]:
+    """
+    Samples a freeform response from the specified model, records the sampling,
+        and returns the sampled text.
+
+    ARGS
+    ====
+    `model_spec`: See `completion_query`.
+    `prompt`: See `completion_query`.
+    `temperature`: Passed to `openai.Completion.create`.
+    `top_p`: Passed to `openai.Completion.create`.
+    `max_tokens`: Passed to `openai.Completion.create`.
+    `stop`: Passed to `openai.Completion.create`.
+    `n_samples`: The number of samples to generate (1 if None).
+    `return_logprobs`: If True, returns the tokens and corresponding logprobs
+        in addition to the sampled text.
+    `kwargs`: See `completion_query`.
+
+    RETURNS
+    =======
+    If `return_logprobs` is True, returns a dict with the sampled text, tokens,
+        and corresponding logprobs. If `n_samples` is None, the outer list is
+        removed from all values.
+    Otherwise, returns the sampled text, or a list of sampled texts if
+        `n_samples` is not None.
+    """
+    response, actual_prompt, metadata = completion_query(
+        prompt=prompt,
+        temperature=temperature,
+        top_p=top_p,
+        max_tokens=max_tokens,
+        stop=stop,
+        n=(1 if n_samples is None else n_samples),
+        model_spec=model_spec,
+        headers={},
+        **kwargs,
+    )
+    sampled = [choice["text"] for choice in response["choices"]]
+    if n_samples is None:
+        sampled = sampled[0]
+    record_sampling(prompt=actual_prompt, sampled=sampled, metadata=metadata)
+
+    if return_logprobs:
+        assert not model_spec.is_chat, "logprobs only works for non-chat models"
+        assert not kwargs.get("logprobs") is None
+
+        def _maybe_tokens(logprobs: Optional[dict]) -> Optional[list[str]]:
+            return logprobs["tokens"] if logprobs is not None else None
+
+        def _maybe_logprobs(logprobs: Optional[dict]) -> Optional[list[float]]:
+            return logprobs["token_logprobs"] if logprobs is not None else None
+
+        def _maybe_top_logprobs(logprobs: Optional[dict]) -> Optional[list[dict[str, float]]]:
+            return [dict(x) for x in logprobs["top_logprobs"]] if logprobs is not None else None
+
+        tokens = [_maybe_tokens(choice["logprobs"]) for choice in response["choices"]]
+        logprobs = [_maybe_logprobs(choice["logprobs"]) for choice in response["choices"]]
+        top_logprobs = [_maybe_top_logprobs(choice["logprobs"]) for choice in response["choices"]]
+        if n_samples is None:
+            tokens = tokens[0]
+            logprobs = logprobs[0]
+            top_logprobs = top_logprobs[0]
+        return {
+            "text": sampled,
+            "tokens": tokens,
+            "logprobs": logprobs,
+            "top_logprobs": top_logprobs,
+        }
+
+    return sampled

evals/evals/base.py

@@ -0,0 +1,153 @@
+"""
+This file defines the base specifications for models, evals, and runs. Running
+evals and most development work should not require familiarity with this file.
+"""
+import base64
+import datetime
+import os
+from typing import TYPE_CHECKING, Any, Dict, Mapping, Optional, Sequence
+
+if TYPE_CHECKING:
+    from dataclasses import dataclass
+else:
+    from pydantic.dataclasses import dataclass
+
+
+@dataclass
+class ModelSpec:
+    """
+    Specification for a model.
+    """
+
+    name: str
+    model: Optional[str] = None
+    api_base: Optional[str] = None
+
+    is_chat: bool = False
+    is_agi: bool = False
+
+    encoding: Optional[str] = None
+    organization: Optional[str] = None
+    api_key: Optional[str] = None
+    extra_options: Optional[Mapping[str, Any]] = None
+    metadata: Optional[Mapping[str, Any]] = None
+    headers: Optional[Mapping[str, Any]] = None
+    strip_completion: bool = True
+    n_ctx: Optional[int] = None
+    format: Optional[str] = None
+    key: Optional[str] = None
+    group: Optional[str] = None
+
+    def __post_init__(self):
+        if self.extra_options is None:
+            self.extra_options = {}
+        if self.headers is None:
+            self.headers = {}
+
+        if self.model is None:
+            raise ValueError(f"Must specify a model")
+
+
+@dataclass
+class BaseEvalSpec:
+    """
+    Specification for a base eval.
+    """
+
+    id: Optional[str] = None
+    metrics: Optional[Sequence[str]] = None
+    description: Optional[str] = None
+    disclaimer: Optional[str] = None
+
+    """
+    True if higher values are better, False if lower values are better.
+    This should really be part of a metric, but it's easier to put it here.
+    """
+    higher_is_better: bool = True
+
+    key: Optional[str] = None
+    group: Optional[str] = None
+
+
+@dataclass
+class EvalSpec:
+    """
+    Specification for an eval.
+    """
+
+    cls: str
+    args: Optional[Dict[str, Any]] = None
+    key: Optional[str] = None
+    group: Optional[str] = None
+
+
+@dataclass
+class EvalSetSpec:
+    """
+    Specification for an eval set.
+    """
+
+    evals: Sequence[str]
+    key: Optional[str] = None
+    group: Optional[str] = None
+
+
+@dataclass
+class ModelSpecs:
+    completions_: Optional[Sequence[ModelSpec]] = None
+    embedding_: Optional[ModelSpec] = None
+    ranking_: Optional[ModelSpec] = None
+
+    @property
+    def embedding(self) -> ModelSpec:
+        if self.embedding_ is None:
+            raise ValueError("Embedding model was not specified")
+        return self.embedding_
+
+    @property
+    def ranking(self) -> ModelSpec:
+        if self.ranking_ is None:
+            raise ValueError("Ranking model was not specified")
+        return self.ranking_
+
+    @property
+    def completion(self) -> ModelSpec:
+        if self.completions_ is None:
+            raise ValueError("Completion model was not specified")
+        return self.completions_[0]
+
+    @property
+    def completions(self) -> Sequence[ModelSpec]:
+        if self.completions_ is None:
+            raise ValueError("Completion model was not specified")
+        return self.completions_
+
+    @property
+    def names(self) -> dict[str, Sequence[str]]:
+        dict = {}
+        if self.completions_ is not None:
+            dict["completions"] = [model.name for model in self.completions_]
+        if self.embedding_ is not None:
+            dict["embedding"] = [self.embedding_.name]
+        if self.ranking_ is not None:
+            dict["ranking"] = [self.ranking_.name]
+        return dict
+
+
+@dataclass
+class RunSpec:
+    model_name: str
+    model_names: dict[str, Sequence[str]]
+    eval_name: str
+    base_eval: str
+    split: str
+    run_config: Dict[str, Any]
+    created_by: str
+    run_id: str = None
+    created_at: str = None
+
+    def __post_init__(self):
+        now = datetime.datetime.utcnow()
+        rand_suffix = base64.b32encode(os.urandom(5)).decode("ascii")
+        self.run_id = now.strftime("%y%m%d%H%M%S") + rand_suffix
+        self.created_at = str(now)

evals/evals/cli/oaieval.py

@@ -0,0 +1,274 @@
+"""
+This file defines the `oaieval` CLI for running evals.
+"""
+import argparse
+import logging
+import shlex
+import sys
+from functools import cached_property
+from typing import Any, Mapping, Optional
+
+import openai
+
+import evals
+import evals.api
+import evals.base
+import evals.record
+from evals.base import ModelSpec, ModelSpecs
+from evals.registry import Registry
+
+logger = logging.getLogger(__name__)
+
+
+def _purple(str):
+    return f"\033[1;35m{str}\033[0m"
+
+
+def get_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(description="Run evals through the API")
+    parser.add_argument("model", type=str, help="Name of a completion model.")
+    parser.add_argument("eval", type=str, help="Name of an eval. See registry.")
+    parser.add_argument("--embedding_model", type=str, default="")
+    parser.add_argument("--ranking_model", type=str, default="")
+    parser.add_argument("--extra_eval_params", type=str, default="")
+    parser.add_argument("--modelspec_extra_options", type=str, default="")
+    parser.add_argument("--max_samples", type=int, default=None)
+    parser.add_argument("--cache", action=argparse.BooleanOptionalAction, default=True)
+    parser.add_argument("--visible", action=argparse.BooleanOptionalAction, default=None)
+    parser.add_argument("--seed", type=int, default=20220722)
+    parser.add_argument("--user", type=str, default="")
+    parser.add_argument("--record_path", type=str, default=None)
+    parser.add_argument(
+        "--log_to_file", type=str, default=None, help="Log to a file instead of stdout"
+    )
+    parser.add_argument("--debug", action=argparse.BooleanOptionalAction, default=False)
+    parser.add_argument("--local-run", action=argparse.BooleanOptionalAction, default=True)
+    parser.add_argument("--dry-run", action=argparse.BooleanOptionalAction, default=False)
+    parser.add_argument("--dry-run-logging", action=argparse.BooleanOptionalAction, default=True)
+    return parser
+
+
+def parse_extra_eval_params(param_str: Optional[str]) -> Mapping[str, Any]:
+    """Parse a string of the form "key1=value1,key2=value2" into a dict."""
+    if not param_str:
+        return {}
+
+    def to_number(x):
+        try:
+            return int(x)
+        except:
+            pass
+        try:
+            return float(x)
+        except:
+            pass
+        return x
+
+    str_dict = dict(kv.split("=") for kv in param_str.split(","))
+    return {k: to_number(v) for k, v in str_dict.items()}
+
+
+def n_ctx_from_model_name(model_name: str) -> Optional[int]:
+    """Returns n_ctx for a given API model name. Model list last updated 2023-03-14."""
+    # note that for most models, the max tokens is n_ctx + 1
+    DICT_OF_N_CTX_BY_MODEL_NAME_PREFIX: dict[str, int] = {
+        "dummy-": 2048,
+        "gpt-3.5-turbo-": 4096,
+        "gpt-4-": 8192,
+        "gpt-4-32k-": 32768,
+        "agi-":128,
+    }
+    DICT_OF_N_CTX_BY_MODEL_NAME: dict[str, int] = {
+        "ada": 2048,
+        "text-ada-001": 2048,
+        "babbage": 2048,
+        "text-babbage-001": 2048,
+        "curie": 2048,
+        "text-curie-001": 2048,
+        "davinci": 2048,
+        "text-davinci-001": 2048,
+        "code-davinci-002": 8000,
+        "text-davinci-002": 4096,
+        "text-davinci-003": 4096,
+        "gpt-3.5-turbo": 4096,
+        "gpt-3.5-turbo-0301": 4096,
+        "gpt-4": 8192,
+        "gpt-4-0314": 8192,
+        "gpt-4-32k": 32768,
+        "gpt-4-32k-0314": 32768,
+        "agi-7B": 128,
+        "agi-13B": 128,
+        "agi-17B": 128,
+        "agi-30B": 128,
+        "agi-65B": 128,
+    }
+    # first, look for a prefix match
+    for model_prefix, n_ctx in DICT_OF_N_CTX_BY_MODEL_NAME_PREFIX.items():
+        if model_name.startswith(model_prefix):
+            return n_ctx
+    # otherwise, look for an exact match and return None if not found
+    return DICT_OF_N_CTX_BY_MODEL_NAME.get(model_name, None)
+
+
+class ModelResolver:
+    # This is a temporary method to identify which models are chat models.
+    # Eventually, the OpenAI API should expose this information directly.
+    CHAT_MODELS = {
+        "gpt-3.5-turbo",
+        "gpt-3.5-turbo-0301",
+        "gpt-4",
+        "gpt-4-0314",
+        "gpt-4-32k",
+        "gpt-4-32k-0314",
+        "dummy-chat",
+        "agi-7B",
+        "agi-13B",
+        "agi-17B",
+        "agi-30B",
+        "agi-65B",        
+    }
+
+    AGI_MODELS = {
+        "agi-7B",
+        "agi-13B",
+        "agi-17B",
+        "agi-30B",
+        "agi-65B",
+    }
+
+    AGI_MODEL_IDS = [model for model in AGI_MODELS]
+
+    DUMMY_MODELS = {
+        "dummy-chat",
+        "dummy-completion",
+    }
+
+    def resolve(self, name: str) -> ModelSpec:
+        if name in self.DUMMY_MODELS:
+            result = ModelSpec(name=name, model=name, is_chat=(name in self.CHAT_MODELS))
+            return result
+
+        if name in self.api_model_ids:
+            result = ModelSpec(
+                name=name,
+                model=name,
+                is_chat=(name in self.CHAT_MODELS),
+                is_agi=(name in self.AGI_MODELS),
+                n_ctx=n_ctx_from_model_name(name),
+            )
+            return result
+
+        raise ValueError(f"Couldn't find model: {name}")
+
+    @cached_property
+    def api_model_ids(self):
+        return([m["id"] for m in openai.Model.list()["data"]] + self.AGI_MODEL_IDS)
+
+
+def run(args, model_resolver: ModelResolver, registry: Optional[Registry] = None):
+    if args.debug:
+        logging.getLogger().setLevel(logging.DEBUG)
+
+    visible = args.visible if args.visible is not None else (args.max_samples is None)
+
+    if args.max_samples is not None:
+        evals.eval.set_max_samples(args.max_samples)
+
+    registry = registry or Registry()
+    eval_spec = registry.get_eval(args.eval)
+    assert (
+        eval_spec is not None
+    ), f"Eval {args.eval} not found. Available: {list(sorted(registry._evals.keys()))}"
+
+    def get_model(name: str) -> ModelSpec:
+        return model_resolver.resolve(name)
+
+    completion_model_specs = [get_model(model) for model in args.model.split(",")]
+
+    for spec in completion_model_specs:
+        spec.extra_options = parse_extra_eval_params(args.modelspec_extra_options)
+
+    model_specs = ModelSpecs(
+        completions_=completion_model_specs,
+        embedding_=get_model(args.embedding_model) if args.embedding_model else None,
+        ranking_=get_model(args.ranking_model) if args.ranking_model else None,
+    )
+
+    run_config = {
+        "model_specs": model_specs,
+        "eval_spec": eval_spec,
+        "seed": args.seed,
+        "max_samples": args.max_samples,
+        "command": " ".join(map(shlex.quote, sys.argv)),
+        "initial_settings": {
+            "visible": visible,
+        },
+    }
+
+    model_name = model_specs.completions_[0].name if len(model_specs.completions_) > 0 else "n/a"
+    eval_name = eval_spec.key
+    run_spec = evals.base.RunSpec(
+        model_name=model_name,
+        model_names=model_specs.names,
+        eval_name=eval_name,
+        base_eval=eval_name.split(".")[0],
+        split=eval_name.split(".")[1],
+        run_config=run_config,
+        created_by=args.user,
+    )
+    if args.record_path is None:
+        record_path = f"/tmp/evallogs/{run_spec.run_id}_{args.model}_{args.eval}.jsonl"
+    else:
+        record_path = args.record_path
+    if args.dry_run:
+        recorder = evals.record.DummyRecorder(run_spec=run_spec, log=args.dry_run_logging)
+    elif args.local_run:
+        recorder = evals.record.LocalRecorder(record_path, run_spec=run_spec)
+    else:
+        recorder = evals.record.Recorder(record_path, run_spec=run_spec)
+
+    api_extra_options = {}
+    if not args.cache:
+        api_extra_options["cache_level"] = 0
+
+    run_url = f"{run_spec.run_id}"
+    logger.info(_purple(f"Run started: {run_url}"))
+
+    extra_eval_params = parse_extra_eval_params(args.extra_eval_params)
+
+    eval_class = registry.get_class(eval_spec)
+    eval = eval_class(
+        model_specs=model_specs,
+        seed=args.seed,
+        name=eval_name,
+        registry=registry,
+        **extra_eval_params,
+    )
+    result = eval.run(recorder)
+    recorder.record_final_report(result)
+
+    if not (args.dry_run or args.local_run):
+        logger.info(_purple(f"Run completed: {run_url}"))
+
+    logger.info("Final report:")
+    for key, value in result.items():
+        logger.info(f"{key}: {value}")
+    return run_spec.run_id
+
+
+def main():
+    parser = get_parser()
+    args = parser.parse_args(sys.argv[1:])
+    logging.basicConfig(
+        format="[%(asctime)s] [%(filename)s:%(lineno)d] %(message)s",
+        level=logging.INFO,
+        filename=args.log_to_file if args.log_to_file else None,
+    )
+    logging.getLogger("openai").setLevel(logging.WARN)
+    if hasattr(openai.error, "set_display_cause"):
+        openai.error.set_display_cause()
+    run(args, model_resolver=ModelResolver())
+
+
+if __name__ == "__main__":
+    main()

evals/evals/cli/oaievalset.py

@@ -0,0 +1,105 @@
+"""
+This file defines the `oaievalset` CLI for running eval sets.
+"""
+import argparse
+import json
+import subprocess
+from pathlib import Path
+from typing import Optional
+
+from evals.registry import Registry
+
+Task = list[str]
+
+
+class Progress:
+    def __init__(self, file: str) -> None:
+        self.file = Path(file)
+        self.completed: list[Task] = []
+
+    def load(self) -> bool:
+        if not self.file.exists():
+            return False
+
+        with self.file.open() as f:
+            for line in f:
+                self.completed.append(json.loads(line))
+        return len(self.completed) > 0
+
+    def add(self, item: Task) -> None:
+        self.completed.append(item)
+        self.save()
+
+    def save(self) -> None:
+        self.file.parent.mkdir(parents=True, exist_ok=True)
+        with self.file.open("w") as f:
+            for item in self.completed:
+                f.write(json.dumps(item) + "\n")
+            print(highlight(f"Saved progress to {self.file}"))
+
+
+def highlight(str: str) -> str:
+    return f"\033[1;32m>>> {str}\033[0m"
+
+
+def get_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(description="Run eval sets through the API")
+    parser.add_argument("model", type=str, help="Name of a completion model.")
+    parser.add_argument("eval_set", type=str, help="Name of eval set. See registry.")
+    parser.add_argument(
+        "--resume",
+        action=argparse.BooleanOptionalAction,
+        default=True,
+        help="Resume from last checkpoint.",
+    )
+    parser.add_argument(
+        "--exit-on-error",
+        action=argparse.BooleanOptionalAction,
+        default=True,
+        help="Exit if any oaieval command fails.",
+    )
+    return parser
+
+
+def run(args, unknown_args, registry: Optional[Registry] = None) -> None:
+    registry = registry or Registry()
+    commands: list[Task] = []
+    eval_set = registry.get_eval_set(args.eval_set)
+    for eval in registry.get_evals(eval_set.evals):
+        command = ["oaieval", args.model, eval.key] + unknown_args
+        if command in commands:
+            continue
+        commands.append(command)
+    num_evals = len(commands)
+
+    progress = Progress(f"/tmp/oaievalset/{args.model}.{args.eval_set}.progress.txt")
+    if args.resume and progress.load():
+        print(f"Loaded progress from {progress.file}")
+        print(f"{len(progress.completed)}/{len(commands)} evals already completed:")
+        for item in progress.completed:
+            print("  " + " ".join(item))
+
+    commands = [c for c in commands if c not in progress.completed]
+    command_strs = [" ".join(cmd) for cmd in commands]
+    print("Going to run the following commands:")
+    for command_str in command_strs:
+        print("  " + command_str)
+
+    num_already_completed = num_evals - len(commands)
+    for idx, command in enumerate(commands):
+        real_idx = idx + num_already_completed
+        print(highlight("Running command: " + " ".join(command) + f" ({real_idx+1}/{num_evals})"))
+        subprocess.run(command, stdout=subprocess.PIPE, check=args.exit_on_error)
+        progress.add(command)
+
+    print(highlight("All done!"))
+
+
+def main() -> None:
+    parser = get_parser()
+    args, unknown_args = parser.parse_known_args()
+    run(args, unknown_args)
+
+
+if __name__ == "__main__":
+    main()

evals/evals/data.py

@@ -0,0 +1,189 @@
+"""
+This file defines utilities for working with data and files of various types.
+"""
+import csv
+import dataclasses
+import gzip
+import itertools
+import json
+import logging
+import os
+import urllib
+from collections.abc import Iterator
+from functools import partial
+from typing import Any, Sequence, Union
+
+import blobfile as bf
+import lz4.frame
+import pydantic
+import pyzstd
+
+logger = logging.getLogger(__name__)
+
+
+def gzip_open(filename: str, mode: str = "rb", openhook: Any = open) -> gzip.GzipFile:
+    """Wrap the given openhook in gzip."""
+    if mode and "b" not in mode:
+        mode += "b"
+
+    return gzip.GzipFile(fileobj=openhook(filename, mode), mode=mode)
+
+
+def lz4_open(filename: str, mode: str = "rb", openhook: Any = open) -> lz4.frame.LZ4FrameFile:
+    if mode and "b" not in mode:
+        mode += "b"
+
+    return lz4.frame.LZ4FrameFile(openhook(filename, mode), mode=mode)
+
+
+def zstd_open(filename: str, mode: str = "rb", openhook: Any = open) -> pyzstd.ZstdFile:
+    if mode and "b" not in mode:
+        mode += "b"
+
+    return pyzstd.ZstdFile(openhook(filename, mode), mode=mode)
+
+
+def open_by_file_pattern(filename: str, mode: str = "r", **kwargs: Any) -> Any:
+    """Can read/write to files on gcs/local with or without gzipping. If file
+    is stored on gcs, streams with blobfile. Otherwise use vanilla python open. If
+    filename endswith gz, then zip/unzip contents on the fly (note that gcs paths and
+    gzip are compatible)"""
+    open_fn = partial(bf.BlobFile, **kwargs)
+    try:
+        if filename.endswith(".gz"):
+            return gzip_open(filename, openhook=open_fn, mode=mode)
+        elif filename.endswith(".lz4"):
+            return lz4_open(filename, openhook=open_fn, mode=mode)
+        elif filename.endswith(".zst"):
+            return zstd_open(filename, openhook=open_fn, mode=mode)
+        else:
+            scheme = urllib.parse.urlparse(filename).scheme
+            if scheme == "" or scheme == "file":
+                return open_fn(
+                    os.path.join(
+                        os.path.dirname(os.path.abspath(__file__)), "registry", "data", filename
+                    ),
+                    mode=mode,
+                )
+            else:
+                return open_fn(filename, mode=mode)
+    except Exception as e:
+        raise RuntimeError(f"Failed to open: {filename}") from e
+
+
+def _get_jsonl_file(path):
+    logger.info(f"Fetching {path}")
+    with open_by_file_pattern(path, mode="r") as f:
+        return list(map(json.loads, f.readlines()))
+
+
+def _get_json_file(path):
+    logger.info(f"Fetching {path}")
+    with open_by_file_pattern(path, mode="r") as f:
+        return json.loads(f.read())
+
+
+def _stream_jsonl_file(path) -> Iterator:
+    logger.info(f"Streaming {path}")
+    with bf.BlobFile(path, "r", streaming=True) as f:
+        for line in f:
+            yield json.loads(line)
+
+
+def get_lines(path) -> list[dict]:
+    """
+    Get a list of lines from a file.
+    """
+    with open_by_file_pattern(path, mode="r") as f:
+        return f.readlines()
+
+
+def get_jsonl(path: str) -> list[dict]:
+    """
+    Extract json lines from the given path.
+    If the path is a directory, look in subpaths recursively.
+
+    Return all lines from all jsonl files as a single list.
+    """
+    if bf.isdir(path):
+        result = []
+        for filename in bf.listdir(path):
+            if filename.endswith(".jsonl"):
+                result += get_jsonl(os.path.join(path, filename))
+        return result
+    return _get_jsonl_file(path)
+
+
+def get_jsonls(paths: Sequence[str], line_limit=None) -> list[dict]:
+    return list(iter_jsonls(paths, line_limit))
+
+
+def get_json(path) -> dict:
+    if bf.isdir(path):
+        raise ValueError("Path is a directory, only files are supported")
+    return _get_json_file(path)
+
+
+def iter_jsonls(paths: Union[str, list[str]], line_limit=None) -> Iterator[dict]:
+    """
+    For each path in the input, iterate over the jsonl files in that path.
+    Look in subdirectories recursively.
+
+    Use an iterator to conserve memory.
+    """
+    if type(paths) == str:
+        paths = [paths]
+
+    def _iter():
+        for path in paths:
+            if bf.isdir(path):
+                for filename in bf.listdir(path):
+                    if filename.endswith(".jsonl"):
+                        yield from iter_jsonls([os.path.join(path, filename)])
+            else:
+                yield from _stream_jsonl_file(path)
+
+    return itertools.islice(_iter(), line_limit)
+
+
+def get_csv(path, fieldnames=None):
+    with bf.BlobFile(path, "r", cache_dir="/tmp/bf_cache", streaming=False) as f:
+        reader = csv.DictReader(f, fieldnames=fieldnames)
+        return [row for row in reader]
+
+
+def _to_py_types(o: Any) -> Any:
+    if isinstance(o, dict):
+        return {k: _to_py_types(v) for k, v in o.items()}
+    if isinstance(o, list):
+        return [_to_py_types(v) for v in o]
+
+    if dataclasses.is_dataclass(o):
+        return _to_py_types(dataclasses.asdict(o))
+
+    # pydantic data classes
+    if isinstance(o, pydantic.BaseModel):
+        return json.loads(o.json())
+
+    return o
+
+
+class EnhancedJSONEncoder(json.JSONEncoder):
+    def default(self, o: Any) -> str:
+        return _to_py_types(o)
+
+
+def jsondumps(o: Any, ensure_ascii: bool = False, **kwargs: Any) -> str:
+    return json.dumps(o, cls=EnhancedJSONEncoder, ensure_ascii=ensure_ascii, **kwargs)
+
+
+def jsondump(o: Any, fp: Any, ensure_ascii: bool = False, **kwargs: Any) -> None:
+    json.dump(o, fp, cls=EnhancedJSONEncoder, ensure_ascii=ensure_ascii, **kwargs)
+
+
+def jsonloads(s: str, **kwargs: Any) -> Any:
+    return json.loads(s, **kwargs)
+
+
+def jsonload(fp: Any, **kwargs: Any) -> Any:
+    return json.load(fp, **kwargs)

evals/evals/elsuite/basic/fuzzy_match.py

@@ -0,0 +1,49 @@
+import evals
+import numpy as np
+from evals.elsuite import utils
+from evals.record import RecorderBase
+
+
+class FuzzyMatch(evals.Eval):
+    def __init__(
+        self,
+        model_specs: evals.ModelSpecs,
+        samples_jsonl: str,
+        *args,
+        max_tokens: int = 500,
+        **kwargs,
+    ):
+        super().__init__(model_specs, *args, **kwargs)
+        self.max_tokens = max_tokens
+        self.samples_jsonl = samples_jsonl
+
+    def eval_sample(self, test_sample, rng):
+        prompt, correct_answers = test_sample["input"], test_sample["ideal"]
+        generated_answer = evals.sample_freeform(
+            self.model_spec,
+            prompt,
+            temperature=0.0,
+            max_tokens=16,
+        )
+        matches = [
+            utils.fuzzy_match(generated_answer, correct_answer)
+            for correct_answer in correct_answers
+        ]
+        evals.record.record_match(
+            True in matches,
+            expected=correct_answers,
+            picked=[generated_answer for i in range(len(correct_answers)) if matches[i]],
+        )
+        evals.record.record_metrics(
+            accuracy=float(True in matches),
+            f1_score=utils.f1_score(generated_answer, correct_answers),
+        )
+
+    def run(self, recorder: RecorderBase):
+        samples = evals.get_jsonl(self.samples_jsonl)
+        self.eval_all_samples(recorder, samples)
+
+        return {
+            "accuracy": np.mean(recorder.get_scores("accuracy")),
+            "f1_score": np.mean(recorder.get_scores("f1_score")),
+        }

evals/evals/elsuite/basic/includes.py

@@ -0,0 +1,38 @@
+from typing import Any
+
+import evals
+import evals.elsuite.utils
+import evals.metrics
+import numpy as np
+
+
+class Includes(evals.Eval):
+    def __init__(
+        self,
+        model_specs: evals.ModelSpecs,
+        samples_jsonl: str,
+        *args,
+        max_tokens: int = 500,
+        **kwargs,
+    ):
+        super().__init__(model_specs, *args, **kwargs)
+        self.max_tokens = max_tokens
+        self.samples_jsonl = samples_jsonl
+
+    def eval_sample(self, sample: Any, *_):
+        sampled = evals.sample_freeform(
+            self.model_spec, sample["input"], max_tokens=self.max_tokens
+        )
+        includes_answer = any(
+            [evals.elsuite.utils.get_answer(sampled, ref) for ref in sample["ideal"]]
+        )
+        evals.record.record_metrics(accuracy=float(includes_answer))
+        return includes_answer
+
+    def run(self, recorder):
+        samples = evals.get_jsonl(self.samples_jsonl)
+        self.eval_all_samples(recorder, samples)
+        events = recorder.get_scores("accuracy")
+        return {
+            "accuracy": np.mean(events),
+        }

evals/evals/elsuite/basic/match.py

@@ -0,0 +1,45 @@
+from typing import Any
+
+import evals
+import evals.metrics
+from evals.prompt.base import is_chat_prompt
+
+
+class Match(evals.Eval):
+    def __init__(
+        self,
+        model_specs: evals.ModelSpecs,
+        samples_jsonl: str,
+        *args,
+        max_tokens: int = 500,
+        num_few_shot: int = 0,
+        few_shot_jsonl: str = None,
+        **kwargs,
+    ):
+        super().__init__(model_specs, *args, **kwargs)
+        self.max_tokens = max_tokens
+        self.samples_jsonl = samples_jsonl
+        self.num_few_shot = num_few_shot
+        if self.num_few_shot > 0:
+            assert few_shot_jsonl is not None, "few shot requires few shot sample dataset"
+            self.few_shot_jsonl = few_shot_jsonl
+            self.few_shot = evals.get_jsonl(self.few_shot_jsonl)
+
+    def eval_sample(self, sample: Any, *_):
+        prompt = sample["input"]
+        if self.num_few_shot > 0:
+            assert is_chat_prompt(sample["input"]), "few shot requires chat prompt"
+            prompt = sample["input"][:-1]
+            for s in self.few_shot[: self.num_few_shot]:
+                prompt += s["sample"]
+            prompt += sample["input"][-1:]
+
+        return evals.check_sampled_text(self.model_spec, prompt, expected=sample["ideal"])
+
+    def run(self, recorder):
+        samples = evals.get_jsonl(self.samples_jsonl)
+        self.eval_all_samples(recorder, samples)
+        events = recorder.get_events("match")
+        return {
+            "accuracy": evals.metrics.get_accuracy(events),
+        }

evals/evals/elsuite/modelgraded/classify.py

@@ -0,0 +1,356 @@
+"""
+Generic eval that uses a prompt + classification.
+"""
+import copy
+import itertools
+import logging
+import string
+from collections import Counter
+from random import Random
+from typing import Callable, Iterable, Optional, Union
+
+import openai
+
+import evals
+import evals.record
+from evals.base import ModelSpec
+from evals.elsuite.utils import PromptFn, format_necessary, scrub_formatting_from_prompt
+
+INVALID_STR = "__invalid__"
+CHOICE_KEY = "choice"
+MATCH_FNS = {
+    "include": lambda x, y: float(x in y),
+    "exact": lambda x, y: float(x == y),
+    "endswith": lambda x, y: x.endswith(y),
+    "starts_or_endswith": lambda x, y: x.startswith(y) or x.endswith(y),
+}
+
+ANSWER_PROMPTS = {
+    # e.g. "Yes"
+    "classify": "Answer the question by printing only a single choice from {choices} (without quotes or punctuation) corresponding to the correct answer with no other text.".strip(),
+    # e.g. "Yes\n The reasons are: ..."
+    "classify_cot": "First, answer by printing a single choice from {choices} (without quotes or punctuation) corresponding to the correct answer. Then, from the next line, explain your reasonings step by step.".strip(),
+    # e.g. "Let's think step by step. ...\nYes"
+    "cot_classify": """
+First, write out in a step by step manner your reasoning to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset. Then print only a single choice from {choices} (without quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the answer by itself on a new line.
+
+Reasoning:""".strip(),
+    "cot_classify_jp": """
+まず、一歩一歩あなたの推論を書き出してください。単に正しい答えを最初に述べることを避けてください。次に、{choices}（引用符や句読点なし）から正しい答えに対応する1つの選択肢を単独の行に書きだしてください。最後に、答えだけを新しい行に繰り返してください。
+
+推論：
+    """.strip(),
+}
+
+
+def choice_to_str(choice_strings: Iterable[str]) -> str:
+    """Return a string of choices, e.g. '"Yes" or "No" or "Maybe"'."""
+    return " or ".join(f'"{choice}"' for choice in choice_strings)
+
+
+def get_choice(text: str, eval_type: str, match_fn: Callable, choice_strings: Iterable[str]) -> str:
+    """Clean the answer string to a choice string to one of choice_strings. Return '__invalid__.' if no match."""
+    lines = text.strip().split("\n")
+    if eval_type.startswith("cot_classify"):
+        lines = lines[::-1]  # reverse lines
+    for line in lines:
+        line = line.strip()
+        line = "".join(c for c in line if c not in string.punctuation)
+        if not line:
+            continue
+        for choice in choice_strings:
+            if match_fn(line, choice):
+                return choice
+    return INVALID_STR
+
+
+def expand_args_dict(args_dict):
+    """Expand a dict of dicts, with namings.
+
+    args_dict = {
+        "a": {"a1": 1, "a2": 2},
+        "b": {"b1": 3, "b2": 4},
+    }
+    expand_args_dict(args_dict) = {
+        "a=a1:b=b1": {"a": ("a1", 1), "b": ("b1", 3)},
+        "a=a1:b=b2": {"a": ("a1", 1), "b": ("b2", 4)},
+    ...}
+    """
+    args_dict = {k: list(v.items()) for k, v in args_dict.items()}
+    keys = list(args_dict.keys())
+    values = list(args_dict.values())
+    new_values = [dict(zip(keys, v)) for v in itertools.product(*values)]
+    new_names = [":".join([f"{k}={v[0]}" for k, v in sorted(d.items())]) for d in new_values]
+    return dict(zip(new_names, new_values))
+
+
+class ModelBasedClassify(evals.Eval):
+    invalid_request_during_completion = 0
+    invalid_request_during_evaluation = 0
+
+    def __init__(
+        self,
+        model_specs: evals.ModelSpecs,
+        samples_jsonl: str,
+        modelgraded_spec: str,
+        *args,
+        match_fn: str = "starts_or_endswith",
+        max_tokens: int = 1024,
+        multicomp_n: Union[int, str] = 1,
+        multicomp_temperature: float = 0.4,
+        samples_renamings: Optional[dict[str, str]] = None,
+        eval_type: Optional[str] = None,
+        metaeval: bool = False,
+        modelgraded_spec_args: Optional[dict[str, dict[str, str]]] = None,
+        **kwargs,
+    ):
+        super().__init__(model_specs, *args, **kwargs)
+        n_models = len(self.model_specs.completions)
+        self.max_tokens = max_tokens
+        self.samples_jsonl = samples_jsonl
+        self.match_fn = MATCH_FNS[match_fn]
+        self.metaeval = metaeval
+        if multicomp_n == "from_models":
+            assert n_models > 1, f"multicomp_n='from_models' but only 1 model is specified."
+            self.multicomp_n = n_models
+        else:
+            assert isinstance(
+                multicomp_n, int
+            ), f"multicomp_n={multicomp_n} must be an int or 'from_models'."
+            self.multicomp_n = multicomp_n
+        self.multicomp_temperature = multicomp_temperature
+        self.samples_renamings = samples_renamings or {}
+
+        # check if multiple models are specified
+        if len(self.model_specs.completions) > 1:
+            assert (
+                self.multicomp_n == n_models
+            ), f"multicomp_n={self.multicomp_n} must be equal to the number of models={len(self.model_specs.completions)} if multiple models are specified."
+
+        if self.model_spec.name == "dummy-completion" or self.model_spec.name == "dummy-chat":
+            self.eval_modelspec = self.model_spec
+        else:
+            self.eval_modelspec = ModelSpec(
+                name="gpt-3.5-turbo", model="gpt-3.5-turbo", is_chat=True
+            )
+
+        """import prompt and set attributes"""
+        modelgraded_specs = self.registry.get_modelgraded_spec(modelgraded_spec)
+        modelgraded_specs = copy.deepcopy(modelgraded_specs)  # since pop() is used
+
+        # 'choice_strings' is a list of strings that specifies the possible choices
+        self.choice_strings = modelgraded_specs.pop("choice_strings")
+        if self.choice_strings == "from_n":
+            self.choice_strings = [str(i + 1) for i in range(self.multicomp_n)]
+        elif self.choice_strings == "from_n_abc":
+            self.choice_strings = [string.ascii_lowercase[i % 26] for i in range(self.multicomp_n)]
+        elif self.choice_strings == "from_n_ABC":
+            self.choice_strings = [string.ascii_uppercase[i % 26] for i in range(self.multicomp_n)]
+        # make sure each choice doesn't contain any punctuation
+        for s in self.choice_strings:
+            assert not any(c in s for c in string.punctuation), f"{s} contains punctuation"
+        #  (optional) 'choice_scores' is a dict that specifies the score for each choice string
+        # if 'choice_scores' is specified, 'scores/' are computed and added to metrics
+        self.choice_scores = modelgraded_specs.pop("choice_scores", {})
+        if self.choice_scores == "from_strings":
+            self.choice_scores = {c: float(c) for c in self.choice_strings}
+        assert all(
+            isinstance(v, (int, float)) for v in self.choice_scores.values()
+        ), f"choice_scores must be a dict of floats, not {self.choice_scores}"
+
+        # (optional) 'eval_type' is a string that specifies the type of classification algorithm
+        #   - "classify": only answer
+        #   - "cot_classify": reason then answer (chain-of-thought) <- most recommended
+        #   - "classify_cot": answer then reason (explanation)
+        # if 'eval_type' is not supplied from modelgraded_specs, then it must be supplied as an argument.
+        #   - Importantly, it also assumes the answer prompt needs to be appended to the prompt.
+        self.eval_type = modelgraded_specs.pop("eval_type", None)
+        if not self.eval_type:
+            append_answer_prompt = True  # append answer prompt to prompt
+            assert eval_type, "eval_type must be specified, in modelgraded_spec or as an argument"
+            self.eval_type = eval_type
+        else:
+            assert (
+                not eval_type
+            ), f"eval_type must be unspecified, if it is specified in modelgraded_spec"
+            append_answer_prompt = False
+
+        # 'prompt' is a string that specifies the model-graded evaluation
+        prompt = modelgraded_specs.pop("prompt")
+        assert isinstance(prompt, str), f"prompt must be a string, not {type(prompt)}"
+        if append_answer_prompt:
+            prompt += "\n\n" + ANSWER_PROMPTS[self.eval_type].format(
+                choices=choice_to_str(self.choice_strings)
+            )
+        self.prompt = [{"role": "user", "content": prompt}]
+
+        # 'input_outputs' is a dict that specifies the input and output keys in the sample
+        # output key is the model's raw response to input key. These are used for filling 'prompt' template.
+        self.input_outputs = modelgraded_specs.pop("input_outputs")
+        assert isinstance(
+            self.input_outputs, dict
+        ), f"input_outputs must be a dict, not {type(self.input_outputs)}"
+
+        # (optional) 'args' is a dict of dicts that specifies additional arguments for 'prompt'
+        # each value in 'args_dict' essentially defines a separate modelgraded classification eval and has own metrics!
+        # if 'modelgraded_spec_args' is specified in eval YAML, it is merged with 'args_dict'
+        self.args_dict = modelgraded_specs.pop("args", {})
+        self.args_dict.update(modelgraded_spec_args or {})
+        if self.args_dict:
+            self.expanded_args_dict = expand_args_dict(self.args_dict)
+        else:
+            self.expanded_args_dict = {}
+
+        # (optional) 'completion_sample_templates'
+        # each key must be one of 'input_outputs'.values(). If 'multicomp_n' > 1, this template is filled 'multicomp_n' times
+        # and the concatenated result is passed to 'prompt' template.
+        self.completion_sample_templates = modelgraded_specs.pop("completion_sample_templates", {})
+        assert all(
+            k in self.input_outputs.values() for k in self.completion_sample_templates
+        ), f"all {self.completion_sample_templates.keys()} must be in {self.input_outputs.values()}, "
+        if self.multicomp_n > 1:
+            assert (
+                self.completion_sample_templates
+            ), "completion_sample_templates must be specified if multicomp_n > 1"
+
+        # since we accept optional args, we need to check that all args are used
+        for key in ("key", "group"):
+            modelgraded_specs.pop(key, None)
+        assert not modelgraded_specs, f"Unused args: {modelgraded_specs}. Typo in YAML?"
+
+    def eval_sample(self, test_sample: dict, rng: Random) -> None:
+        """Evaluate a single sample.
+
+        Recorded metrics are always: one of the self.choice_strings, or "__invalid__".
+        """
+        if self.samples_renamings:
+            test_sample = {self.samples_renamings.get(k, k): v for k, v in test_sample.items()}
+        if self.multicomp_n > 1:
+            test_sample["n"] = self.multicomp_n
+        completions = {}
+        if self.metaeval:
+            # assert outputs exist in the data
+            for v in self.input_outputs.values():
+                assert v in test_sample, f"Missing output '{v}' in sample {test_sample.keys()}"
+                completions[v] = test_sample[v]
+        # remove outputs from the data
+        test_sample = {
+            k: v for k, v in test_sample.items() if k not in list(self.input_outputs.values())
+        }
+        for k in self.input_outputs:
+            test_sample[k] = scrub_formatting_from_prompt(test_sample[k])
+
+        if not self.metaeval:
+            try:
+                for k, v in self.input_outputs.items():
+                    if self.multicomp_n > 1 and v in self.completion_sample_templates:
+                        completion = ""
+                        completion_i_template = self.completion_sample_templates[v]
+                        for i in range(self.multicomp_n):
+                            if len(self.model_specs.completions) > 1:
+                                # use a separate model for each completion
+                                model_spec = self.model_specs.completions[i]
+                            else:
+                                # use the single model for all completions
+                                model_spec = self.model_spec
+                            get_input_completion = PromptFn(
+                                test_sample[k],
+                                model_spec=model_spec,
+                                max_tokens=self.max_tokens,
+                                temperature=self.multicomp_temperature,
+                            )
+                            completion_i, _ = get_input_completion()
+                            completion += format_necessary(
+                                completion_i_template,
+                                i=i + 1,
+                                i_abc=string.ascii_lowercase[i % 26],
+                                i_ABC=string.ascii_uppercase[i % 26],
+                                output=completion_i,
+                                n=self.multicomp_n,
+                            )
+                    else:
+                        get_input_completion = PromptFn(
+                            test_sample[k],
+                            model_spec=self.model_spec,
+                            max_tokens=self.max_tokens,
+                        )
+                        completion, _ = get_input_completion()
+                    completions[v] = completion
+            except openai.error.InvalidRequestError:
+                self.invalid_request_during_completion += 1
+                return
+
+        try:
+            metrics = {}
+            evaluate = PromptFn(
+                self.prompt,
+                model_spec=self.eval_modelspec,
+                max_tokens=self.max_tokens,
+            )
+            eval_kwargs = dict(**completions, **test_sample)
+            if self.expanded_args_dict and len(self.expanded_args_dict) > 1:
+                args_dict = self.expanded_args_dict
+            elif self.expanded_args_dict and len(self.expanded_args_dict) == 1:
+                # if there is only one combination, don't bother with the metric name
+                args_dict = {CHOICE_KEY: v for v in self.expanded_args_dict.values()}
+            else:
+                args_dict = {CHOICE_KEY: {}}
+            for metric, args in args_dict.items():
+                args = {k: v[1] for k, v in args.items()}
+                evaluation, _ = evaluate(**args, **eval_kwargs)
+                choice = get_choice(evaluation, self.eval_type, self.match_fn, self.choice_strings)
+                if choice == INVALID_STR:
+                    logging.warn(
+                        f"Choices {self.choice_strings} not parsable for {self.eval_type}: {evaluation}"
+                    )
+                metrics[metric] = choice
+                if self.metaeval:
+                    assert (
+                        metric in test_sample
+                    ), f"Missing label for metric '{metric}' in sample {test_sample.keys()}"
+                    metrics[metric + "_metascore"] = choice == test_sample[metric]
+
+        except openai.error.InvalidRequestError:
+            self.invalid_request_during_evaluation += 1
+            return
+
+        evals.record.record_metrics(**metrics)
+
+        return choice
+
+    def run(self, recorder):
+        samples = evals.get_jsonl(self.samples_jsonl)
+
+        self.eval_all_samples(recorder, samples)
+        all_sample_metrics = recorder.get_metrics()
+
+        record_metrics = {}
+        if self.expanded_args_dict and len(self.expanded_args_dict) > 1:
+            metrics = sorted(self.expanded_args_dict)
+        else:
+            metrics = [CHOICE_KEY]
+        for metric in metrics:
+            chosen = [m[metric] for m in all_sample_metrics if metric in m]
+            # if there is a best choice, compute the score
+            if self.choice_scores:
+                # assumption: each INVALID_STR contributes the lowest score
+                lowest_score = min(self.choice_scores.values())
+                scores = [
+                    self.choice_scores[choice] if choice != INVALID_STR else lowest_score
+                    for choice in chosen
+                ]
+                record_metrics[f"score/{metric}"] = sum(scores) / len(all_sample_metrics)
+            # compute the counts and ratios
+            counts = dict(Counter(chosen))
+            missing_samples = len(all_sample_metrics) - len(chosen)
+            if missing_samples:
+                counts["__missing_samples__"] = missing_samples
+            record_metrics.update({f"counts/{metric}/{k}": v for k, v in counts.items()})
+            if self.metaeval:
+                metascores = [m[metric + "_metascore"] for m in all_sample_metrics if metric in m]
+                record_metrics[f"metascore/{metric}"] = sum(metascores) / len(all_sample_metrics)
+
+        record_metrics["invalid_request_during_completion"] = self.invalid_request_during_completion
+        record_metrics["invalid_request_during_evaluation"] = self.invalid_request_during_evaluation
+
+        return record_metrics

evals/evals/elsuite/translate.py

@@ -0,0 +1,75 @@
+from typing import Any
+
+from sacrebleu.metrics.bleu import BLEU
+
+import evals
+import evals.metrics
+from evals.prompt.base import is_chat_prompt
+
+
+class Translate(evals.Eval):
+    def __init__(
+        self,
+        model_specs: evals.ModelSpecs,
+        samples_jsonl: str,
+        *args,
+        max_tokens: int = 500,
+        num_few_shot: int = 0,
+        few_shot_jsonl: str = None,
+        **kwargs,
+    ):
+        super().__init__(model_specs, *args, **kwargs)
+        self.max_tokens = max_tokens
+        self.samples_jsonl = samples_jsonl
+
+        self.num_few_shot = num_few_shot
+        if self.num_few_shot > 0:
+            assert few_shot_jsonl is not None, "few shot requires few shot sample dataset"
+            self.few_shot_jsonl = few_shot_jsonl
+            self.few_shot = evals.get_jsonl(self.few_shot_jsonl)
+
+        self.bleu = BLEU(effective_order=True)
+
+    def eval_sample(self, sample: Any, *_):
+        prompt = sample["input"]
+        expected = sample["ideal"]
+        if self.num_few_shot > 0:
+            assert is_chat_prompt(sample["input"]), "few shot requires chat prompt"
+            prompt = sample["input"][:-1]
+            for s in self.few_shot[: self.num_few_shot]:
+                prompt += s["sample"]
+            prompt += sample["input"][-1:]
+
+        if isinstance(expected, tuple):
+            expected = list(expected)
+        elif not isinstance(expected, list):
+            expected = [expected]
+
+        sampled = evals.sample_freeform(self.model_spec, prompt, max_tokens=self.max_tokens)
+
+        score = None
+        if expected is not None:
+            score = self.bleu.sentence_score(sampled, expected).score
+            evals.record.record_metrics(sacrebleu_sentence_score=score)
+
+            match = score > 30
+
+            if score is not None:
+                evals.record.record_match(
+                    match, expected=expected, sampled=sampled, sacrebleu_sentence_score=score
+                )
+            return match
+
+    def run(self, recorder):
+        samples = evals.get_jsonl(self.samples_jsonl)
+        self.eval_all_samples(recorder, samples)
+        events = recorder.get_events("match")
+
+        sampled = list(map(lambda e: e.data["sampled"], events))
+        expected = list(map(lambda e: e.data["expected"], events))
+        sacrebleu_score = BLEU().corpus_score(sampled, [expected]).score
+
+        return {
+            "accuracy": evals.metrics.get_accuracy(events),
+            "sacrebleu_score": sacrebleu_score,
+        }

evals/evals/elsuite/utils.py

@@ -0,0 +1,140 @@
+import copy
+import re
+import string
+from collections import Counter, defaultdict
+
+from evals.api import sample_freeform
+from evals.prompt.base import chat_prompt_to_text_prompt, is_chat_prompt
+
+
+def get_answer(text, answer_prompt):
+    idx = text.rfind(answer_prompt)
+    if idx == -1:
+        return None
+    return text[idx + len(answer_prompt) :]
+
+
+def get_consensus(answers):
+    counts = defaultdict(int)
+    for answer in answers:
+        counts[answer] += 1
+    counts[None] = 0
+    return max(counts, key=counts.get)
+
+
+def normalize(s: str) -> str:
+    """Lower text and remove punctuation, articles and extra whitespace."""
+    s = s.split("\n")[0]
+    s = s.lower()
+    exclude = set(string.punctuation)
+    s = "".join(char for char in s if char not in exclude)
+    s = re.sub(r"\b(a|an|the)\b", " ", s)
+    s = " ".join(s.split())
+    return s
+
+
+def fuzzy_match(s1: str, s2: str) -> bool:
+    s1 = normalize(s1)
+    s2 = normalize(s2)
+
+    if s1 == "" or s2 == "":
+        return s1 == s2
+
+    return s1 in s2 or s2 in s1
+
+
+def get_scores_from_text(text: str) -> dict:
+    pattern = r"## (.+?)\n.+?(\d)/5"
+    matches = re.findall(pattern, text, re.DOTALL)
+    return {k: int(v) for k, v in dict(matches).items()}
+
+
+def get_yesno_from_text(text: str) -> dict:
+    pattern = r"## (.+?)\n.+?([yn])"
+    matches = re.findall(pattern, text, re.DOTALL)
+    return {k: v for k, v in dict(matches).items()}
+
+
+def get_letter_from_data(data: str) -> str:
+    last_y = (data.rfind("y"), "y")
+    last_n = (data.rfind("n"), "n")
+    char = max(last_y, last_n)[1]
+    return char
+
+
+def f1_score(prediction: str, answers: list[str]) -> float:
+    def _f1_score(prediction: str, ground_truth: str):
+        prediction_tokens = normalize(prediction).split()
+        ground_truth_tokens = normalize(ground_truth).split()
+        common = Counter(prediction_tokens) & Counter(ground_truth_tokens)
+        num_same = sum(common.values())
+        if num_same == 0:
+            return 0
+        precision = 1.0 * num_same / len(prediction_tokens)
+        recall = 1.0 * num_same / len(ground_truth_tokens)
+        f1 = (2 * precision * recall) / (precision + recall)
+        return f1
+
+    return max([_f1_score(prediction, answer) for answer in answers])
+
+
+def scrub_formatting_from_prompt(prompt):
+    scrubbed_prompt = copy.copy(prompt)
+
+    if is_chat_prompt(prompt):
+        for i, msg in enumerate(scrubbed_prompt):
+            if "content" in msg:
+                scrubbed_prompt[i]["content"] = msg["content"].replace("{", "{{").replace("}", "}}")
+    else:
+        scrubbed_prompt = scrubbed_prompt.replace("{", "{{").replace("}", "}}")
+    return scrubbed_prompt
+
+
+def format_necessary(template: str, **kwargs: dict[str, str]) -> str:
+    """Format a template string with only necessary kwargs."""
+    keys = [k[1] for k in string.Formatter().parse(template) if k[1]]
+    assert all(k in kwargs for k in keys), f"Required: {keys}, got: {sorted(kwargs)}"
+    cur_keys = {k: kwargs[k] for k in keys}
+    return template.format(**cur_keys)
+
+
+class PromptFn:
+    """Wrap calls to model with prompt"""
+
+    def __init__(self, prompt, model_spec, max_tokens, temperature=0, completion_kwargs=None):
+        self.prompt = prompt
+        self.max_tokens = max_tokens
+        self.model_spec = model_spec
+        self.temperature = temperature
+        self.completion_kwargs = completion_kwargs or {}
+
+    def __call__(self, **kwargs):
+        # if any input kwargs is chat prompt, convert to text prompt
+        kwargs = {
+            k: chat_prompt_to_text_prompt(v, render_for_completion=False)
+            if is_chat_prompt(v)
+            else v
+            for k, v in kwargs.items()
+        }
+        if is_chat_prompt(self.prompt):
+            prompt = []
+            for msg in self.prompt:
+                formatted_msg = copy.copy(msg)
+                if "content" in formatted_msg:
+                    formatted_msg["content"] = format_necessary(formatted_msg["content"], **kwargs)
+                prompt.append(formatted_msg)
+        else:
+            # Prompt is a string
+            prompt = format_necessary(self.prompt, **kwargs)
+
+        completion = sample_freeform(
+            self.model_spec,
+            prompt,
+            max_tokens=self.max_tokens,
+            temperature=self.temperature,
+            top_p=1,
+            frequency_penalty=0,
+            presence_penalty=0,
+            **self.completion_kwargs,
+        )
+        return completion, prompt

evals/evals/eval.py

@@ -0,0 +1,155 @@
+"""
+This file defines the base class for evals.
+"""
+import abc
+import asyncio
+import concurrent.futures
+import logging
+import os
+import random
+from multiprocessing.pool import ThreadPool
+from typing import Any, Awaitable, Callable, Dict, List, Optional, Tuple
+
+from tqdm import tqdm
+
+from .base import ModelSpec, ModelSpecs
+from .record import RecorderBase
+from .registry import Registry
+
+logger = logging.getLogger(__name__)
+
+
+SHUFFLE_SEED = 123
+_MAX_SAMPLES = None
+
+
+def _index_samples(samples: List[Any]) -> List[Tuple[Any, int]]:
+    """Shuffle `samples` and pair each sample with its index."""
+    indices = list(range(len(samples)))
+    random.Random(SHUFFLE_SEED).shuffle(indices)
+    if _MAX_SAMPLES is not None:
+        indices = indices[:_MAX_SAMPLES]
+    logger.info(f"Evaluating {len(indices)} samples")
+    work_items = [(samples[i], i) for i in indices]
+    return work_items
+
+
+def set_max_samples(max_samples: int):
+    global _MAX_SAMPLES
+    _MAX_SAMPLES = max_samples
+
+
+class Eval(abc.ABC):
+    """
+    Evaluation classes generally should override two methods:
+    `eval_sample`: Takes in a test sample and a random number generator and
+        records the metrics of interest.
+    `run`: Takes in a recorder and runs the evaluation. Generally, most `run`
+        methods will follow this same pattern: loading the data, calling
+        `eval_all_samples`, and aggregating the recorded results.
+    """
+
+    def __init__(
+        self,
+        model_specs: ModelSpecs,
+        seed: int = 20220722,
+        name: str = "no_name_eval.default",
+        registry: Optional[Registry] = None,
+    ):
+        splits = name.split(".")
+        if len(splits) < 2:
+            raise ValueError(f"Eval name must at least have <base_eval>.<split>. Got name {name}")
+
+        self.model_specs = model_specs
+        self.seed = seed
+        self.name = name
+        self.registry = registry or Registry()
+
+    def eval_sample(self, sample: Any, rng: random.Random):
+        raise NotImplementedError()
+
+    @classmethod
+    def create_and_run(cls, model_specs: ModelSpecs, *args, **kwargs) -> Dict[str, float]:
+        logging.info(f"Running {cls.__name__} with {model_specs}, args: {args}, kwargs: {kwargs}")
+        return cls(model_specs).run(*args, **kwargs)
+
+    @property
+    def model_spec(self) -> ModelSpec:
+        """Helper for more ergonomic access to a single model."""
+        return self.model_specs.completion
+
+    @abc.abstractmethod
+    def run(self, recorder: RecorderBase) -> Dict[str, float]:
+        """Run the evaluation with the corresponding recorder."""
+        raise NotImplementedError()
+
+    async def async_eval_all_samples(
+        self,
+        eval_fn: Callable[[Tuple[Any, int]], Awaitable[Tuple[int, Any]]],
+        samples: List[Any],
+        concurrency: int = 32,
+        show_progress: bool = True,
+    ):
+        work_items = _index_samples(samples)
+        semaphore = asyncio.Semaphore(concurrency)
+
+        async def eval_fn_with_semaphore(args):
+            async with semaphore:
+                return await eval_fn(args)
+
+        futures = [asyncio.ensure_future(eval_fn_with_semaphore(args)) for args in work_items]
+
+        for future in tqdm(
+            asyncio.as_completed(futures), total=len(samples), disable=not show_progress
+        ):
+            await future
+
+    def eval_all_samples(
+        self,
+        recorder: RecorderBase,
+        samples,
+        show_progress=True,
+    ):
+        """
+        Evaluate all provided samples in parallel.
+        """
+        work_items = _index_samples(samples)
+        threads = int(os.environ.get("EVALS_THREADS", "10"))
+        show_progress = bool(os.environ.get("EVALS_SHOW_EVAL_PROGRESS", show_progress))
+        timeout = float(os.environ.get("EVALS_THREAD_TIMEOUT", "40"))
+
+        def eval_sample(args):
+            """
+            Evaluate a single sample.
+            """
+            sample, idx = args
+            base_name, split = self.name.split(".")[0:2]
+            sample_id = f"{base_name}.{split}.{idx}"
+            with recorder.as_default_recorder(sample_id):
+                recorder.record_raw(sample)
+                seed = f"{sample_id}:{self.seed}".encode("utf-8")
+                rng = random.Random(seed)
+                return idx, self.eval_sample(sample, rng)
+
+        def worker_thread(args):
+            """
+            Worker thread for evaluating a single sample.
+            """
+            while True:
+                executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
+                future = executor.submit(eval_sample, args=args)
+                try:
+                    result = future.result(timeout=timeout)
+                    return result
+                except concurrent.futures.TimeoutError as e:
+                    executor.shutdown(wait=False)
+
+        with ThreadPool(threads) as pool:
+            if os.environ.get("EVALS_SEQUENTIAL", "0") in {"1", "true", "yes"}:
+                logger.info(f"Running in sequential mode!")
+                iter = map(eval_sample, work_items)
+            else:
+                logger.info(f"Running in threaded mode with {threads} threads!")
+                iter = pool.imap_unordered(worker_thread, work_items)
+            idx_and_result = list(tqdm(iter, total=len(work_items), disable=not show_progress))
+        return [r for _, r in sorted(idx_and_result)]

evals/evals/formatting.py

@@ -0,0 +1,34 @@
+"""
+This file defines utilities for adding multiple choice questions to prompts.
+"""
+import random
+from typing import Optional
+
+
+def make_abc(answers, *, correct_idx=0, shuffle=True, rng: Optional[random.Random] = None):
+    """
+    ARGS
+    ====
+    `answers`: A sequence of strings, each of which is an answer choice.
+    `correct_idx`: The integer index of the correct answer.
+    `shuffle`: If True, shuffle the answer choices in the returned string.
+    `rng`: If `shuffle` is True, this is the random number generator to use.
+
+    RETURNS
+    =======
+    A tuple of (options, correct_answer) where `options` is a string of
+    newline-separated answer choices (e.g., "A) blah") and `correct_answer` is
+    the correct answer as a single character (e.g., "A").
+    """
+
+    p = list(range(len(answers)))
+    if shuffle:
+        if rng is None:
+            raise ValueError("shuffle=True requires rng")
+        rng.shuffle(p)
+    options = ""
+    for i, j in enumerate(p):
+        if i > 0:
+            options += "\n"
+        options += chr(ord("A") + i) + ") " + answers[j]
+    return options, chr(ord("A") + p.index(correct_idx))

evals/evals/metrics.py

@@ -0,0 +1,76 @@
+"""
+This file defines various common metrics of interest.
+"""
+import random
+from typing import Optional, Sequence, Set
+
+import numpy as np
+
+from evals.record import Event
+
+
+def get_accuracy(events: Sequence[Event]) -> float:
+    num_correct = 0
+    num_total = 0
+    for event in events:
+        num_total += 1
+        num_correct += int(event.data["correct"])
+    if num_total == 0:
+        return float("nan")
+    else:
+        return num_correct / num_total
+
+
+def get_bootstrap_accuracy_std(events: Sequence[Event], num_samples: int = 1000):
+    vals = [m.data["correct"] for m in events]
+    return np.std([np.mean(random.sample(vals, len(vals) // 2)) for _ in range(1000)])
+
+
+def get_confusion_matrix(
+    matches: Sequence[Event], class_labels: Optional[Set] = None
+) -> np.ndarray:
+    labels = set()
+    for match in matches:
+        labels.add(match.data["expected"])
+    if class_labels is None:
+        labels = {label: i for i, label in enumerate(sorted(labels))}
+    else:
+        assert labels.issubset(class_labels)
+        labels = {label: i for i, label in enumerate(class_labels)}
+    result = np.zeros((len(labels), len(labels) + 1), dtype=int)
+    for match in matches:
+        i = labels[match.data["expected"]]
+        j = labels.get(match.data["picked"], len(labels))
+        result[i, j] += 1
+    return result
+
+
+def compute_matthew_corr(confusion_matrix):
+    assert confusion_matrix.shape == (2, 3), f"Got shape: {confusion_matrix.shape}"
+    r = confusion_matrix[:, :2]
+    r[:, 0] += confusion_matrix[:, 2]
+    return (r[1, 1] * r[0, 0] - r[1, 0] * r[0, 1]) / np.sqrt(
+        r[1, :].sum() * r[0, :].sum() * r[:, 0].sum() * r[:, 1].sum()
+    )
+
+
+def compute_precision(confusion_matrix, idx=0):
+    return confusion_matrix[idx, idx] / confusion_matrix[:, idx].sum()
+
+
+def compute_recall(confusion_matrix, idx=0):
+    return confusion_matrix[idx, idx] / confusion_matrix[idx, :].sum()
+
+
+def compute_f_score(confusion_matrix, idx=0, beta=1.0):
+    precision = compute_precision(confusion_matrix, idx=idx)
+    recall = compute_recall(confusion_matrix, idx=idx)
+    return (1 + beta**2) * (precision * recall) / (beta**2 * precision + recall)
+
+
+def compute_averaged_f_score(confusion_matrix, beta=1.0, average="macro"):
+    assert average in ["macro"]
+    f_scores = []
+    for i in range(confusion_matrix.shape[0]):
+        f_scores.append(compute_f_score(confusion_matrix, idx=i, beta=beta))
+    return np.array(f_scores).mean()

evals/evals/prompt/base.py

@@ -0,0 +1,118 @@
+"""
+This file defines the classes for how to manage prompts for different types of
+models, i.e., "chat models" vs. "non chat models".
+"""
+import logging
+import threading
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from typing import Dict, List, Union
+
+logger = logging.getLogger(__name__)
+ENCODER_LOCK = threading.Lock()
+
+# This is an approximation to the type accepted as the `prompt` field to `openai.Completion.create` calls
+OpenAICreatePrompt = Union[str, list[str], list[int], list[list[int]]]
+
+# This is the type accepted as the `prompt` field to `openai.ChatCompletion.create` calls
+OpenAIChatMessage = Dict[str, str]  # A message is a dictionary with "role" and "content" keys
+OpenAICreateChatPrompt = List[OpenAIChatMessage]  # A chat log is a list of messages
+
+
+def chat_prompt_to_text_prompt(
+    prompt: OpenAICreateChatPrompt, render_for_completion: bool = True
+) -> str:
+    """
+    Render a chat prompt as a text prompt. User and assistant messages are separated by newlines
+    and prefixed with "User: " and "Assistant: ", respectively, unless there is only one message.
+    System messages have no prefix.
+    """
+    assert is_chat_prompt(prompt), f"Expected a chat prompt, got {prompt}"
+    chat_to_prefixes = {
+        # roles
+        "system": "",
+        # names
+        "example_user": "User: ",
+        "example_assistant": "Assistant: ",
+    }
+
+    # For a single message, be it system, user, or assistant, just return the message
+    if len(prompt) == 1:
+        return prompt[0]["content"]
+
+    text = ""
+    for msg in prompt:
+        role = msg["name"] if "name" in msg else msg["role"]
+        prefix = chat_to_prefixes.get(role, role.capitalize() + ": ")
+        content = msg["content"]
+        text += f"{prefix}{content}\n"
+    if render_for_completion:
+        text += "Assistant: "
+    return text.lstrip()
+
+
+def text_prompt_to_chat_prompt(prompt: str, role: str = "system") -> OpenAICreateChatPrompt:
+    assert isinstance(prompt, str), f"Expected a text prompt, got {prompt}"
+    return [
+        {"role": role, "content": prompt},
+    ]
+
+
+@dataclass
+class Prompt(ABC):
+    """
+    A `Prompt` encapsulates everything required to present the `raw_prompt` in different formats,
+    e.g., a normal unadorned format vs. a chat format.
+    """
+
+    @abstractmethod
+    def to_openai_create_prompt(self):
+        """
+        Return the actual data to be passed as the `prompt` field to either `openai.ChatCompletion.create`,
+        if the model is a chat model, or `openai.Completion.create` otherwise.
+        See the above types to see what each API call is able to handle.
+        """
+
+
+def is_chat_prompt(prompt: Prompt) -> bool:
+    return isinstance(prompt, list) and all(isinstance(msg, dict) for msg in prompt)
+
+
+@dataclass
+class CompletionPrompt(Prompt):
+    """
+    A `Prompt` object that wraps prompts to be compatible with non chat models, which use `openai.Completion.create`.
+    """
+
+    raw_prompt: Union[OpenAICreatePrompt, OpenAICreateChatPrompt]
+
+    def _render_chat_prompt_as_text(self, prompt: OpenAICreateChatPrompt) -> OpenAICreatePrompt:
+        return chat_prompt_to_text_prompt(prompt)
+
+    def to_openai_create_prompt(self) -> OpenAICreatePrompt:
+        if is_chat_prompt(self.raw_prompt):
+            return self._render_chat_prompt_as_text(self.raw_prompt)
+        return self.raw_prompt
+
+
+@dataclass
+class ChatCompletionPrompt(Prompt):
+    """
+    A `Prompt` object that wraps prompts to be compatible with chat models, which use `openai.ChatCompletion.create`.
+
+    The format expected by chat models is a list of messages, where each message is a dict with "role" and "content" keys.
+    """
+
+    raw_prompt: Union[OpenAICreatePrompt, OpenAICreateChatPrompt]
+
+    def _render_text_as_chat_prompt(self, prompt: str) -> OpenAICreateChatPrompt:
+        """
+        Render a text string as a chat prompt. The default option we adopt here is to simply take the full prompt
+        and treat it as a system message.
+        """
+        return text_prompt_to_chat_prompt(prompt)
+
+    def to_openai_create_prompt(self) -> OpenAICreateChatPrompt:
+        if is_chat_prompt(self.raw_prompt):
+            return self.raw_prompt
+        return self._render_text_as_chat_prompt(self.raw_prompt)

evals/evals/record.py

@@ -0,0 +1,480 @@
+"""
+This file defines the recorder classes which log eval results in different ways,
+such as to a local JSON file or to a remote Snowflake database.
+
+If you would like to implement a custom recorder, you can see how the
+`LocalRecorder` and `Recorder` classes inherit from the `RecorderBase` class and
+override certain methods.
+"""
+import atexit
+import contextlib
+import dataclasses
+import logging
+import threading
+import time
+from contextvars import ContextVar
+from datetime import datetime, timezone
+from typing import Any, List, Optional, Sequence
+
+import blobfile as bf
+
+import evals
+from evals.base import RunSpec
+from evals.data import jsondumps
+from evals.utils.misc import t
+from evals.utils.snowflake import SnowflakeConnection
+
+logger = logging.getLogger(__name__)
+
+MIN_FLUSH_EVENTS = 100
+MAX_SNOWFLAKE_BYTES = 16 * 10**6
+MIN_FLUSH_SECONDS = 10
+
+_default_recorder: ContextVar[Optional["RecorderBase"]] = ContextVar(
+    "default_recorder", default=None
+)
+
+
+def default_recorder() -> Optional["RecorderBase"]:
+    return _default_recorder.get()
+
+
+@dataclasses.dataclass
+class Event:
+    run_id: str
+    event_id: int
+    sample_id: Optional[str]
+    type: str
+    data: dict
+    created_by: str
+    created_at: str
+
+
+class RecorderBase:
+    """
+    The standard events for which recording methods are provided are:
+    - `match`: A match or non match, as specified by the `correct` bool, between
+        the `expected` and `picked` results.
+    - `embedding`: An embedding of the `prompt` of type `embedding_type`.
+    - `sampling`: What was `sampled` from the model given the input `prompt`.
+    - `cond_logp`: The conditional log probability, as `logp`, of the
+        `completion` from the model given the input `prompt`.
+    - `pick_option`: The option `picked` by the model out of the valid `options`
+        given the input `prompt`.
+    - `raw`: A raw sample specified by the `data`.
+    - `metrics`: A set of metrics specified by the `kwargs`.
+    - `error`: An `error` along with an accompanying `msg`.
+    - `extra`: Any extra `data` of interest to be recorded.
+    For these events, helper methods are defined at the bottom of this file.
+    More generally, you can record any event by calling `record_event` with the
+    event `type` and `data`.
+    Finally, you can also record a final report using `record_final_report`.
+    """
+
+    def __init__(
+        self,
+        run_spec: evals.base.RunSpec,
+    ) -> None:
+        self._sample_id: ContextVar[Optional[int]] = ContextVar("_sample_id", default=None)
+        self.run_spec = run_spec
+        self._events: List[Event] = []
+        self._last_flush_time = time.time()
+        self._flushes_done = 0
+        self._written_events = 0
+        self._flushes_started = 0
+        self._event_lock = threading.Lock()
+        atexit.register(self.flush_events)
+
+    @contextlib.contextmanager
+    def as_default_recorder(self, sample_id: str):
+        sample_id_token = self._sample_id.set(sample_id)
+        default_recorder_token = _default_recorder.set(self)
+        yield
+        _default_recorder.reset(default_recorder_token)
+        self._sample_id.reset(sample_id_token)
+
+    def current_sample_id(self) -> Optional[str]:
+        return self._sample_id.get()
+
+    def get_events(self, type: str) -> Sequence[Event]:
+        with self._event_lock:
+            return [event for event in self._events if event.type == type]
+
+    def get_metrics(self):
+        return list(map(lambda x: x.data, self.get_events("metrics")))
+
+    def get_scores(self, key: str):
+        return list(map(lambda e: e.data[key], self.get_events("metrics")))
+
+    def _create_event(self, type, data=None, sample_id=None):
+        if sample_id is None:
+            sample_id = self.current_sample_id()
+        if sample_id is None:
+            raise ValueError("No sample_id set! Either pass it in or use as_default_recorder!")
+
+        return Event(
+            run_id=self.run_spec.run_id,
+            event_id=len(self._events),
+            type=type,
+            sample_id=sample_id,
+            data=data,
+            created_by=self.run_spec.created_by,
+            created_at=str(datetime.now(timezone.utc)),
+        )
+
+    def _flush_events_internal(self, events_to_write: Sequence[Event]):
+        pass
+
+    def flush_events(self):
+        with self._event_lock:
+            if len(self._events) == self._written_events:
+                return
+            events_to_write = self._events[self._written_events :]
+            self._written_events = len(self._events)
+            self._flushes_started += 1
+        self._flush_events_internal(events_to_write)
+
+    def record_event(self, type, data=None, sample_id=None):
+        if sample_id is None:
+            sample_id = self.current_sample_id()
+        if sample_id is None:
+            raise ValueError("No sample_id set! Either pass it in or use as_default_recorder!")
+
+        with self._event_lock:
+            event = Event(
+                run_id=self.run_spec.run_id,
+                event_id=len(self._events),
+                type=type,
+                sample_id=sample_id,
+                data=data,
+                created_by=self.run_spec.created_by,
+                created_at=str(datetime.now(timezone.utc)),
+            )
+            self._events.append(event)
+            if (
+                self._flushes_done < self._flushes_started
+                or len(self._events) < self._written_events + MIN_FLUSH_EVENTS
+                or time.time() < self._last_flush_time + MIN_FLUSH_SECONDS
+            ):
+                return
+            events_to_write = self._events[self._written_events :]
+            self._written_events = len(self._events)
+            self._flushes_started += 1
+            self._flush_events_internal(events_to_write)
+
+    def record_match(self, correct: bool, *, expected=None, picked=None, sample_id=None, **extra):
+        assert isinstance(
+            correct, bool
+        ), f"correct must be a bool, but was a {type(correct)}: {correct}"
+
+        if isinstance(expected, list) and len(expected) == 1:
+            expected = expected[0]
+        data = {
+            "correct": bool(correct),
+            "expected": expected,
+            "picked": picked,
+            **extra,
+        }
+        self.record_event("match", data, sample_id=sample_id)
+
+    def record_embedding(self, prompt, embedding_type, sample_id=None, **extra):
+        data = {
+            "prompt": prompt,
+            "embedding_type": embedding_type,
+            **extra,
+        }
+        self.record_event("embedding", data, sample_id=sample_id)
+
+    def record_sampling(self, prompt, sampled, sample_id=None, **extra):
+        data = {
+            "prompt": prompt,
+            "sampled": sampled,
+            **extra,
+        }
+        self.record_event("sampling", data, sample_id=sample_id)
+
+    def record_cond_logp(self, prompt, completion, logp, sample_id=None, **extra):
+        data = {
+            "prompt": prompt,
+            "completion": completion,
+            "logp": logp,
+            **extra,
+        }
+        self.record_event("cond_logp", data, sample_id=sample_id)
+
+    def record_pick_option(self, prompt, options, picked, sample_id=None, **extra):
+        data = {
+            "prompt": prompt,
+            "options": options,
+            "picked": picked,
+            **extra,
+        }
+        self.record_event("pick_option", data, sample_id=sample_id)
+
+    def record_raw(self, data):
+        self.record_event("raw_sample", data)
+
+    def record_metrics(self, **kwargs):
+        self.record_event("metrics", kwargs)
+
+    def record_error(self, msg: str, error: Exception, **kwargs):
+        data = {
+            "type": type(error).__name__,
+            "message": str(error),
+        }
+        data.update(kwargs)
+        self.record_event("error", data)
+
+    def record_extra(self, data, sample_id=None):
+        self.record_event("extra", data, sample_id=sample_id)
+
+    def record_final_report(self, final_report: Any):
+        logging.info(f"Final report: {final_report}. Not writing anywhere.")
+
+
+def _green(str):
+    return f"\033[1;32m{str}\033[0m"
+
+
+def _red(str):
+    return f"\033[1;31m{str}\033[0m"
+
+
+class DummyRecorder(RecorderBase):
+    """
+    A "recorder" which only logs certain events to the console.
+    Can be used by passing `--dry-run` when invoking `oaieval`.
+    """
+
+    def __init__(self, run_spec: RunSpec, log: bool = True):
+        super().__init__(run_spec)
+        self.log = log
+
+    def record_event(self, type, data, sample_id=None):
+        from evals.registry import registry
+
+        if self.run_spec is None:
+            return
+
+        base_eval_spec = registry.get_base_eval(self.run_spec.base_eval)
+        if base_eval_spec and len(base_eval_spec.metrics) >= 1:
+            primary_metric = base_eval_spec.metrics[0]
+        else:
+            primary_metric = "accuracy"
+
+        with self._event_lock:
+            event = self._create_event(type, data)
+            self._events.append(event)
+
+        msg = f"Not recording event: {event}"
+
+        if type == "match":
+            accuracy_good = (
+                primary_metric == "accuracy" or primary_metric.startswith("pass@")
+            ) and (data.get("correct", False) or data.get("accuracy", 0) > 0.5)
+            f1_score_good = primary_metric == "f1_score" and data.get("f1_score", 0) > 0.5
+            if accuracy_good or f1_score_good:
+                msg = _green(msg)
+            else:
+                msg = _red(msg)
+
+        if self.log:
+            logging.info(msg)
+
+
+class LocalRecorder(RecorderBase):
+    """
+    A recorder which logs events to the specified JSON file.
+    This is the default recorder used by `oaieval`.
+    """
+
+    def __init__(self, log_path: Optional[str], run_spec: RunSpec):
+        super().__init__(run_spec)
+        self.event_file_path = log_path
+        if log_path is not None:
+            with bf.BlobFile(log_path, "wb") as f:
+                f.write((jsondumps({"spec": dataclasses.asdict(run_spec)}) + "\n").encode("utf-8"))
+
+    def _flush_events_internal(self, events_to_write: Sequence[Event]):
+        start = time.time()
+        try:
+            lines = [jsondumps(event) + "\n" for event in events_to_write]
+        except TypeError as e:
+            logger.error(f"Failed to serialize events: {events_to_write}")
+            raise e
+
+        with bf.BlobFile(self.event_file_path, "ab") as f:
+            f.write(b"".join([l.encode("utf-8") for l in lines]))
+
+        logger.info(
+            f"Logged {len(lines)} rows of events to {self.event_file_path}: insert_time={t(time.time()-start)}"
+        )
+
+        self._last_flush_time = time.time()
+        self._flushes_done += 1
+
+    def record_final_report(self, final_report: Any):
+        with bf.BlobFile(self.event_file_path, "ab") as f:
+            f.write((jsondumps({"final_report": final_report}) + "\n").encode("utf-8"))
+
+        logging.info(f"Final report: {final_report}. Logged to {self.event_file_path}")
+
+
+class Recorder(RecorderBase):
+    """
+    A recorder which logs events to Snowflake.
+    Can be used by passing `--no-local-run` when invoking `oaieval`.
+    """
+
+    def __init__(
+        self,
+        log_path: Optional[str],
+        run_spec: evals.base.RunSpec,
+        snowflake_connection: Optional[SnowflakeConnection] = None,
+    ) -> None:
+        super().__init__(run_spec)
+        self.event_file_path = log_path
+        self._writing_lock = threading.Lock()
+
+        if snowflake_connection is None:
+            snowflake_connection = SnowflakeConnection()
+        self._conn = snowflake_connection
+
+        if log_path is not None:
+            with bf.BlobFile(log_path, "wb") as f:
+                f.write((jsondumps({"spec": dataclasses.asdict(run_spec)}) + "\n").encode("utf-8"))
+
+        query = """
+            INSERT ALL INTO runs (run_id, model_name, eval_name, base_eval, split, run_config, settings, created_by, created_at)
+            VALUES (%(run_id)s, %(model_name)s, %(eval_name)s, %(base_eval)s, %(split)s, run_config, settings, %(created_by)s, %(created_at)s)
+            SELECT PARSE_JSON(%(run_config)s) AS run_config, PARSE_JSON(%(settings)s) AS settings
+        """
+        self._conn.robust_query(
+            command=query,
+            params={
+                "run_id": run_spec.run_id,
+                "model_name": jsondumps(run_spec.model_names),
+                "eval_name": run_spec.eval_name,
+                "base_eval": run_spec.base_eval,
+                "split": run_spec.split,
+                "run_config": jsondumps(run_spec.run_config),
+                "settings": jsondumps(run_spec.run_config.get("initial_settings", {})),
+                "created_by": run_spec.created_by,
+                "created_at": run_spec.created_at,
+            },
+        )
+        atexit.register(self.flush_events)
+
+    def _flush_events_internal(self, events_to_write: Sequence[Event]):
+        with self._writing_lock:
+            try:
+                lines = [jsondumps(event) + "\n" for event in events_to_write]
+            except TypeError as e:
+                logger.error(f"Failed to serialize events: {events_to_write}")
+                raise e
+            idx_l = 0
+            while idx_l < len(events_to_write):
+                total_bytes = 0
+                idx_r = idx_l
+                while (
+                    idx_r < len(events_to_write)
+                    and total_bytes + len(lines[idx_r]) < MAX_SNOWFLAKE_BYTES
+                ):
+                    total_bytes += len(lines[idx_r])
+                    idx_r += 1
+                assert idx_r > idx_l
+                start = time.time()
+                buffer = [
+                    (
+                        event.run_id,
+                        event.event_id,
+                        event.sample_id,
+                        event.type,
+                        jsondumps(event.data),
+                        event.created_by,
+                        event.created_at,
+                    )
+                    for event in events_to_write[idx_l:idx_r]
+                ]
+                query = """
+                INSERT INTO events (run_id, event_id, sample_id, type, data, created_by, created_at)
+                SELECT Column1 AS run_id, Column2 as event_id, Column3 AS sample_id, Column4 AS type, PARSE_JSON(Column5) AS data, Column6 AS created_by, Column7 AS created_at
+                FROM VALUES(%s, %s, %s, %s, %s, %s, %s)
+                """
+                self._conn.robust_query(command=query, seqparams=buffer, many=True)
+                logger.info(
+                    f"Logged {len(buffer)} rows of events to Snowflake: insert_time={t(time.time()-start)}"
+                )
+                idx_l = idx_r
+
+            with bf.BlobFile(self.event_file_path, "ab") as f:
+                f.write(b"".join([l.encode("utf-8") for l in lines]))
+            self._last_flush_time = time.time()
+            self._flushes_done += 1
+
+    def record_final_report(self, final_report: Any):
+        with self._writing_lock:
+            with bf.BlobFile(self.event_file_path, "ab") as f:
+                f.write((jsondumps({"final_report": final_report}) + "\n").encode("utf-8"))
+            query = """
+                UPDATE runs
+                SET final_report = PARSE_JSON(%(final_report)s)
+                WHERE run_id = %(run_id)s
+            """
+            self._conn.robust_query(
+                command=query,
+                params={
+                    "run_id": self.run_spec.run_id,
+                    "final_report": jsondumps(final_report),
+                },
+            )
+
+    def record_event(self, type, data=None, sample_id=None):
+        # try to serialize data so we fail early!
+        _ = jsondumps(data)
+        return super().record_event(type, data, sample_id)
+
+
+#########################################################################
+### Helper methods which use the thread local global default recorder ###
+#########################################################################
+
+
+def current_sample_id() -> str:
+    return default_recorder().current_sample_id
+
+
+def record_match(correct: bool, *, expected=None, picked=None, **extra):
+    return default_recorder().record_match(correct, expected=expected, picked=picked, **extra)
+
+
+def record_embedding(prompt, embedding_type, **extra):
+    return default_recorder().record_embedding(prompt, embedding_type, **extra)
+
+
+def record_sampling(prompt, sampled, **extra):
+    return default_recorder().record_sampling(prompt, sampled, **extra)
+
+
+def record_cond_logp(prompt, completion, logp, **extra):
+    return default_recorder().record_cond_logp(prompt, completion, logp, **extra)
+
+
+def record_pick_option(prompt, options, picked, **extra):
+    return default_recorder().record_pick_option(prompt, options, picked, **extra)
+
+
+def record_raw(data):
+    return default_recorder().record_raw(data)
+
+
+def record_metrics(**extra):
+    return default_recorder().record_metrics(**extra)
+
+
+def record_error(msg: str, error: Exception = None, **extra):
+    return default_recorder().record_error(msg, error, **extra)
+
+
+def record_extra(data):
+    return default_recorder().record_extra(data)

evals/evals/registry.py

@@ -0,0 +1,174 @@
+"""
+Functions to handle registration of evals. To add a new eval to the registry,
+add an entry in one of the YAML files in the `../registry` dir.
+By convention, every eval name should start with {base_eval}.{split}.
+"""
+
+import difflib
+import functools
+import logging
+import os
+import re
+from functools import partial
+from pathlib import Path
+from typing import Any, Iterator, Sequence, Type, Union
+
+import yaml
+
+from evals.base import BaseEvalSpec, EvalSetSpec, EvalSpec
+from evals.utils.misc import make_object
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_PATHS = [Path(__file__).parents[0].resolve() / "registry", Path.home() / ".evals"]
+
+
+class Registry:
+    def __init__(self, registry_paths: Sequence[Union[str, Path]] = DEFAULT_PATHS):
+        self._registry_paths = [Path(p) if isinstance(p, str) else p for p in registry_paths]
+
+    def make_callable(self, spec):
+        return partial(make_object(spec.cls).create_and_run, **(spec.args or {}))
+
+    def get_class(self, spec: dict) -> Any:
+        return make_object(spec.cls, **(spec.args if spec.args else {}))
+
+    def _dereference(self, name: str, d: dict, object: str, type: Type) -> dict:
+        if not name in d:
+            return None
+
+        def get_alias():
+            if isinstance(d[name], str):
+                return d[name]
+            if isinstance(d[name], dict) and "id" in d[name]:
+                return d[name]["id"]
+            return None
+
+        logger.debug(f"Looking for {name}")
+        while True:
+            alias = get_alias()
+
+            if alias is None:
+                break
+            name = alias
+
+        spec = d[name]
+
+        try:
+            return type(**spec)
+        except TypeError as e:
+            raise TypeError(f"Error while processing {object} {name}: {e}")
+
+    def get_modelgraded_spec(self, name: str) -> dict[str, Any]:
+        assert name in self._modelgraded_specs, (
+            f"Modelgraded spec {name} not found. "
+            f"Closest matches: {difflib.get_close_matches(name, self._modelgraded_specs.keys(), n=5)}"
+        )
+        return self._modelgraded_specs[name]
+
+    def get_eval(self, name: str) -> EvalSpec:
+        return self._dereference(name, self._evals, "eval", EvalSpec)
+
+    def get_eval_set(self, name: str) -> EvalSetSpec:
+        return self._dereference(name, self._eval_sets, "eval set", EvalSetSpec)
+
+    def get_evals(self, patterns: Sequence[str]) -> Iterator[EvalSpec]:
+        # valid patterns: hello, hello.dev*, hello.dev.*-v1
+        def get_regexp(pattern):
+            pattern = pattern.replace(".", "\\.")
+            pattern = pattern.replace("*", ".*")
+            return re.compile(f"^{pattern}$")
+
+        regexps = list(map(get_regexp, patterns))
+        for name in self._evals:
+            # if any regexps match, return the name
+            if any(map(lambda regexp: regexp.match(name), regexps)):
+                yield self.get_eval(name)
+
+    def get_base_evals(self) -> list[BaseEvalSpec]:
+        base_evals = []
+        for name, spec in self._evals.items():
+            if name.count(".") == 0:
+                base_evals.append(self.get_base_eval(name))
+        return base_evals
+
+    def get_base_eval(self, name: str) -> BaseEvalSpec:
+        if not name in self._evals:
+            return None
+
+        spec_or_alias = self._evals[name]
+        if isinstance(spec_or_alias, dict):
+            spec = spec_or_alias
+            try:
+                return BaseEvalSpec(**spec)
+            except TypeError as e:
+                raise TypeError(f"Error while processing base eval {name}: {e}")
+
+        alias = spec_or_alias
+        return BaseEvalSpec(id=alias)
+
+    def _process_file(self, registry, path):
+        with open(path, "r") as f:
+            d = yaml.safe_load(f)
+
+        if d is None:
+            # no entries in the file
+            return
+
+        for name, spec in d.items():
+            assert name not in registry, f"duplicate entry: {name} from {path}"
+            if isinstance(spec, dict):
+                if "key" in spec:
+                    raise ValueError(
+                        f"key is a reserved keyword, but was used in {name} from {path}"
+                    )
+                if "group" in spec:
+                    raise ValueError(
+                        f"group is a reserved keyword, but was used in {name} from {path}"
+                    )
+                if "cls" in spec:
+                    raise ValueError(
+                        f"cls is a reserved keyword, but was used in {name} from {path}"
+                    )
+
+                spec["key"] = name
+                spec["group"] = str(os.path.basename(path).split(".")[0])
+                if "class" in spec:
+                    spec["cls"] = spec["class"]
+                    del spec["class"]
+            registry[name] = spec
+
+    def _process_directory(self, registry, path):
+        files = Path(path).glob("*.yaml")
+        for file in files:
+            self._process_file(registry, file)
+
+    def _load_registry(self, paths):
+        """Load registry from a list of paths.
+
+        Each path or yaml specifies a dictionary of name -> spec.
+        """
+        registry = {}
+        for path in paths:
+            logging.info(f"Loading registry from {path}")
+            if os.path.exists(path):
+                if os.path.isdir(path):
+                    self._process_directory(registry, path)
+                else:
+                    self._process_file(registry, path)
+        return registry
+
+    @functools.cached_property
+    def _eval_sets(self):
+        return self._load_registry([p / "eval_sets" for p in self._registry_paths])
+
+    @functools.cached_property
+    def _evals(self):
+        return self._load_registry([p / "evals" for p in self._registry_paths])
+
+    @functools.cached_property
+    def _modelgraded_specs(self):
+        return self._load_registry([p / "modelgraded" for p in self._registry_paths])
+
+
+registry = Registry()

evals/evals/registry/data/README.md

@@ -0,0 +1,44 @@
+### Registry Data
+
+The JSONL need to be pulled via `git-lfs` / downloaded to view.
+
+Here are some example JSONLs for reference and how they are used in evals. See our [eval templates docs](../../../docs/eval-templates.md) for more details.
+
+`test_match/samples.jsonl` In the associated eval from [`test-basic.yaml`](../evals/test-basic.yaml), we see this data is used in a `Match` class, which means we will check if a completion starts with the value for "ideal" key.
+```json
+{"input": [{"role": "system", "content": "Complete the phrase as concisely as possible."}, {"role": "user", "content": "Once upon a "}], "ideal": "time"}
+{"input": [{"role": "system", "content": "Complete the phrase as concisely as possible."}, {"role": "user", "content": "The first US president was "}], "ideal": "George Washington"}
+{"input": [{"role": "system", "content": "Complete the phrase as concisely as possible."}, {"role": "user", "content": "OpenAI was founded in 20"}], "ideal": "15"}
+```
+Another example of a Match eval is:
+```json
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Spell this sentence backwards, character by character: We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests."}], "ideal": ".stseuqer etairporppani tcejer dna ,sesimerp tcerrocni egnellahc ,sekatsim sti timda ,snoitseuq puwollof rewsna ot TPGtahC rof elbissop ti sekam tamrof eugolaid ehT .yaw lanoitasrevnoc a ni stcaretni hcihw TPGtahC dellac ledom a deniart ev’eW"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Spell this sentence backwards, character by character: Latencies will vary over time so we recommend benchmarking prior to making deployment decisions"}], "ideal": "snoisiced tnemyolped gnikam ot roirp gnikramhcneb dnemmocer ew os emit revo yrav lliw seicnetaL"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Spell this sentence backwards, character by character: Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity."}], "ideal": ".ytinamuh fo lla stifeneb—snamuh naht retrams yllareneg era taht smetsys IA—ecnegilletni lareneg laicifitra taht erusne ot si noissim ruO"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Spell this sentence backwards, character by character: There are several things we think are important to do now to prepare for AGI."}], "ideal": ".IGA rof eraperp ot won od ot tnatropmi era kniht ew sgniht lareves era erehT"}
+```
+
+`test_fuzzy_match/samples.jsonl` In the associated eval from [`test-basic.yaml`](../evals/test-basic.yaml), we see this data is used in a `FuzzyMatch` class, which means we will check if a completion includes a normalized version of the "ideal" key or vice-versa.
+```json
+{"input": [{"role": "system", "content": "Answer the following questions as concisely as possible."}, {"role": "system", "content": "What's the capital of France?", "name": "example_user"}, {"role": "system", "content": "Paris", "name": "example_assistant"}, {"role": "system", "content": "What's 2+2?", "name": "example_user"}, {"role": "system", "content": "4", "name": "example_assistant"}, {"role": "user", "content": "Who is the girl who plays eleven in stranger things?"}], "ideal": ["Millie Bobby Brown"]}
+{"input": [{"role": "system", "content": "Answer the following questions as concisely as possible."}, {"role": "system", "content": "What's the capital of France?", "name": "example_user"}, {"role": "system", "content": "Paris", "name": "example_assistant"}, {"role": "system", "content": "What's 2+2?", "name": "example_user"}, {"role": "system", "content": "4", "name": "example_assistant"}, {"role": "user", "content": "What season did derek die in grey's?"}], "ideal": ["Season 11", "11"]}
+{"input": [{"role": "system", "content": "Answer the following questions as concisely as possible."}, {"role": "system", "content": "What's the capital of France?", "name": "example_user"}, {"role": "system", "content": "Paris", "name": "example_assistant"}, {"role": "system", "content": "What's 2+2?", "name": "example_user"}, {"role": "system", "content": "4", "name": "example_assistant"}, {"role": "user", "content": "Who played the girl elf in the hobbit?"}], "ideal": ["Evangeline Lilly"]}
+```
+
+`logic/samples.jsonl` In the associated eval from [`test-basic.yaml`](../evals/logic.yaml), we see this data is used with a `ModelBasedClassify` class with [`fact`](../modelgraded/fact.yaml) model-graded yaml, which will compare the factual content of the completion against a ground truth.
+```json
+{"input":[{"role":"system","content":"Solve the following logical puzzle. Carefully think step by step, and show your reasoning. If there is not enough information to solve the puzzle, conclude with 'There is not enough information.' There are five students, Anna, Bob and Cynthia, Dan and Eliza. They all tell the truth. Anna is taller than Cynthia. Bob says he's taller than Anna if and only if Eliza is the shortest. Cynthia is taller than Dan. Eliza is shorter than Dan. Who's the tallest in the group? Let's think step by step:"}],"ideal":"Anna > Cynthia > Dan > Eliza. But, based on Bob's statement, there are still two possibilities: 1. Bob is taller than Eliza, making Eliza the shortest, making Bob taller than Anna, making Bob the tallest. 2. Bob is shorter than Eliza: this would still be valid, as Eliza wouldn't be the shortest and therefore Bob isn't taller than Anna. And Anna would be the tallest. So there's not enough information"}
+{"input":[{"role":"system","content":"Laura thinks that Jessica thinks that Angie is only 23 years old. Angie thinks Josie knows where Laura's mother is. Jessica thinks Laura was once an engineer. Josie thinks Laura is friendly. Based on the text, what thoughts do we know that Laura, Jessica, Angie, and Josie have?"}],"ideal":"Laura thinks: Jessica thinks Angie is only 23 years old. Jessica thinks: Laura was once an engineer. Angie thinks: Josie knows where Laura's mother is. Josie thinks: Laura is friendly."}
+{"input":[{"role":"system","content":"At a party, there are 100 people. Some always lie and some always tell the truth. They all know which one of them is a truth-teller and which one is a liar. After the party, you ask each person how many truth-tellers they shook hands with. Each person gives a different answer, ranging from 0 to 99. How many were truth-tellers and how many were liars?"}],"ideal":"There is 1 truth-teller and 99 liars at the party."}
+{"input":[{"role":"system","content":"Two people want to cross a river. The only way to get across is with a boat that they find on one side; but that boat can only take one person at a time. The boat cannot return on its own, and there are no ropes to haul it back, yet both persons manage to cross using the boat. How did they do it?"}],"ideal":"The people are on different sides of the river, so the person on the same side as the boat originally can cross first to bring the boat to the side with the other person, then that person can cross."}
+{"input":[{"role":"system","content":"There are two men. One of them is wearing a red shirt, and the other is wearing a blue shirt. The two men are named Andrew and Bob, but we do not know which is Andrew and which is Bob. The guy in the blue shirt says, 'I am Andrew.' The guy in the red shirt says, 'I am Bob.' If we know that at least one of them lied, then what color shirt is Andrew wearing?"}],"ideal":"Andrew is wearing the red shirt."}
+{"input":[{"role":"system","content":"Which word does NOT belong with the others? A. index B. glossary C. chapter D. book"}],"ideal":"D. book"}
+{"input":[{"role":"system","content":"The day before yesterday, Chris was 7 years old. Next year he'll turn 10. How is this possible?"}],"ideal":"Assuming today is January 1st of any given year: Two days ago, on December 30th, Chris was 7 years old. On December 31st, Chris celebrated his 8th birthday. On December 31st of this year, Chris will celebrate his 9th birthday. By the end of this year, on December 31st, Chris will be 10 years old. Therefore, on January 1st of the following year, Chris will be 10 years old."}
+{"input":[{"role":"system","content":"Inhabitants of an island lie consistently on Tuesdays, Thursdays, and Saturdays, and they tell the truth on the other four days of the week. You have forgotten what day of the week it is, so you ask a passerby. 'Saturday,' he answers. 'And what day will it be tomorrow?' you inquire. 'Wednesday,' he replies. Can you tell what day it is today?"}],"ideal":"Based on answer 1, today cannot be M, W, F, Su, or Sa (lying day). Based on answer 2, today cannot be M, W, F, Su, or Tu (lying day). So, today must be Thursday."}
+{"input":[{"role":"system","content":"You are on an island populated by two tribes. Members of one tribe consistently lie. Members of the other tribe always tell the truth. Tribe members can recognize one another, but you can't tell them apart. You meet two people, C and D on the island. C says, 'Exactly one of us is from the liars tribe.' Which tribe is D from?"}],"ideal":"D is from the Liars tribe."}
+{"input":[{"role":"system","content":"There are five people in a room. Each person will either always tell the truth or always tell a lie. Each person is asked the following question: How many liars are among you? The answers are: \"one\", \"two\", \"three\", \"four\", \"five\". How many liars are in the room?"}],"ideal":"There are four liars."}
+```
+
+### Dataset attributions
+
+This work includes data from the Illinois Intentional Tort Qualitative Dataset, which was compiled by the Qualitative Reasoning Group at Northwestern University. The dataset is freely available under the Creative Commons Attribution 4.0 license from https://www.qrg.northwestern.edu/Resources/caselawcorpus.html

evals/evals/registry/data/aba_mrpc_true_false/samples.jsonl

@@ -0,0 +1,110 @@
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer with general experience not considered competent to handle a case involving a specialized field of law."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Newly admitted lawyers cannot be as competent as practitioners with long experience."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In an emergency, a lawyer is allowed to provide assistance in a matter even if they do not have the ordinarily required skill."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must always prepare for legal problems with which they are not familiar by seeking special training."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer cannot accept representation if they do not currently possess the required level of competence."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney may consult with lawyers outside of the attorney's firm on the client's case without telling the client."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney may consult with lawyers inside of the attorney's firm on the client's case without telling the client."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When lawyers from more than one law firm are providing legal services to a client on a particular matter, they do not need to consult with each other and the client about the scope of their respective representations."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Maintaining competence does not require a lawyer to keep abreast of changes in the law and its practice."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer cannot ethically represent a client if they disagree with the client's political, economic, social or moral views related to the matter."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may limit the scope of representation if it is reasonable and the client gives informed consent."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In a criminal case, the lawyer has the final say in decisions regarding the client's plea, whether to waive jury trial, and whether the client will testify."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can counsel a client to engage in conduct that the lawyer knows is criminal or fraudulent."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can counsel a client regarding the consequences of conduct that the lawyer knows is criminal or fraudulent."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer's representation of a client does not imply endorsement of the client's views or activities."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to draft and deliver documents that they know are fraudulent in order to assist their client."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When a lawyer discovers that their client's ongoing conduct is criminal or fraudulent, the lawyer must report the client to the authorities."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may provide limited representation to a client as long as the limitation is reasonable and the client is properly informed."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is prohibited from discussing the legal consequences of a proposed course of conduct with a client if that conduct is clearly criminal."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "If a lawyer knows that a client expects assistance not permitted by the Rules of Professional Conduct, the lawyer must immediately withdraw from the representation."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to charge a contingent fee in a domestic relations matter, where the payment is contingent upon the securing of a divorce or the amount of alimony or support, when the client is unable to pay otherwise."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must communicate the scope of representation and the basis or rate of the fee and expenses to the client in writing before the representation commences in all cases."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may charge a client an unreasonable fee or an unreasonable amount for expenses, as long as the client agrees to it."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to enter into a fee arrangement that might induce the lawyer to curtail services for the client or perform them in a way contrary to the client's interest."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A contingent fee agreement must be orally agreed upon between the lawyer and client, stating the method by which the fee is to be determined."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may accept property in payment for services, such as an ownership interest in an enterprise, without any limitations or restrictions."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may divide a fee with another lawyer who is not in the same firm, even if the client does not agree to the arrangement."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is required to return any unearned portion of a fee if the client fires them without warning."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may charge a contingent fee for representing a defendant in a criminal case."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When determining the reasonableness of a fee, factors such as the time and labor required, the novelty and difficulty of the questions involved, and the skill requisite to perform the legal service properly should be considered."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to reveal information relating to the representation of a client if they believe it is necessary to prevent the client from committing a crime.."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to reveal information relating to the representation of a client if they believe it is necessary to prevent the client from committing a crime that may result in financial injury to another person."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is required to reveal information relating to the representation of a client if it is necessary to prevent the client from committing a crime that may result in financial injury to another person."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may reveal information relating to the representation of a client to detect and resolve conflicts of interest arising from changes in the composition or ownership of a firm, even if the revealed information would compromise the attorney-client privilege."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may disclose information relating to the representation of a client if the disclosure is impliedly authorized to carry out the representation."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is prohibited from revealing any information relating to the representation of a client, regardless of the circumstances."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may disclose information relating to the representation of a client to secure legal advice about the lawyer's compliance with the Rules of Professional Conduct."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to reveal information relating to the representation of a client to establish a defense in a controversy between the lawyer and the client."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must disclose unprotected information relating to the representation of a client, even if the disclosure could reasonably lead to the discovery of protected information by a third party."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to disclose information relating to a client's representation to detect and resolve conflicts of interest without the client's consent."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must take reasonable efforts to prevent unauthorized access to or disclosure of client information, even if the lawyer is not at fault for the unauthorized access or disclosure."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is not required to take any special security measures when transmitting client information if the method of communication affords a reasonable expectation of privacy."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "The duty of confidentiality ends when the client-lawyer relationship is terminated."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must always appeal an adverse ruling regarding disclosure of client information, regardless of the client's wishes."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to disclose client information if a court or other tribunal orders the disclosure, even without the client's informed consent."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may represent a client with a concurrent conflict of interest without obtaining informed consent from each affected client, as long as the lawyer reasonably believes they can provide competent and diligent representation."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to represent a client if the representation involves asserting a claim by one client against another client represented by the same lawyer in the same litigation."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may represent clients with economically adverse interests in unrelated matters without obtaining their informed consent."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer's duty of loyalty and independent judgment can be materially limited by responsibilities to a former client."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to represent a client in a transaction where the lawyer's own conduct is in question, as long as the lawyer can provide competent and diligent representation."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may engage in sexual relationships with a client if the relationship predates the formation of the client-lawyer relationship."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who is closely related by blood or marriage to another lawyer may represent a client in a matter where the other lawyer represents another party, without obtaining informed consent from each client."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may accept payment from a source other than the client if the client is informed and consents, and the arrangement does not compromise the lawyer's duty of loyalty or independent judgment."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A parent who pays for the lawyer's services on behalf of a child may direct or control some of the legal strategy decisions made."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "In all cases, clients can consent to a representation where a conflict of interest exists."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "Representation is prohibited if the lawyer cannot reasonably conclude that they will be able to provide competent and diligent representation."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "Informed consent requires the client to be aware of the possible effects on loyalty, confidentiality, and the attorney-client privilege when multiple clients are represented in a single matter."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A client who has given consent to a conflict cannot revoke the consent or terminate the lawyer's representation at any time."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A client who has given consent to a conflict cannot revoke the consent or terminate the lawyer's representation if the lawyer reasonably and honestly believes the timing would harm the client's interests."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "General and open-ended advance consent to future conflicts is considered effective."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A lawyer may not take inconsistent legal positions in different tribunals at different times on behalf of different clients."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A lawyer is required to obtain informed consent from a client, confirmed in writing, when there is a potential conflict of interest."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is prohibited from representing multiple parties to a negotiation if their interests are fundamentally antagonistic to each other."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can represent multiple clients with generally aligned interests even if there are some differences in interest among them."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must always maintain impartiality between commonly represented clients."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "As between commonly represented clients, the attorney-client privilege does not attach."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer representing an organization also represents all of its affiliated organizations, such as parent and subsidiary companies."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who is a member of a corporation's board of directors must resign from the board or cease acting as the corporation's lawyer when a conflict of interest arises."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In common representation, if one client asks the lawyer not to disclose information relevant to the representation to the other client, the lawyer must withdraw from representing both clients."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer representing multiple clients in the same matter should consider the potential additional cost, embarrassment, and recrimination if the common representation fails."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When seeking to establish or adjust a relationship between clients, the lawyer's role is that of partisanship normally expected in other circumstances."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In a law firm, a lawyer's disqualification due to a personal interest will result in the disqualification of all other lawyers in the firm."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who moves from one firm to another can be screened from participation in a matter, and the new firm can represent a client with adverse interests without obtaining the former client's informed consent."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A law firm is prohibited from representing a client with interests adverse to those of a client represented by a formerly associated lawyer."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A disqualification prescribed by Rule 1.10 may be waived by the affected client under the conditions stated in Rule 1.7."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer's disqualification based on prior work as a law student will result in the disqualification of all other lawyers in the firm."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney is allowed to touch and move contraband on behalf of the client."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In ex parte proceedings, an attorney is not required to reveal information that may be harmful to their client's case."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A prosecutor is obligated to timely disclose favorable evidence to the defense, even if it is inadmissible or has no impact on the outcome of the case."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is acceptable for an attorney to communicate directly with a person who is represented by counsel on a specific matter without the consent of their counsel."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A prosecutor has a duty to protect the accused's right to counsel."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can make false statements of fact to adversaries and third parties."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney must not act with the sole purpose of delaying, burdening, or embarrassing other parties while obtaining evidence."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Lawyers are allowed to make out-of-court statements that are completely true, but that they reasonably should know have a substantial likelihood of materially prejudicing the case."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is acceptable for a prosecutor to make true comments that have a substantial likelihood of heightening public condemnation of the accused."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must self-report when they know they have violated the rules of professional conduct."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Attorneys have an affirmative duty to expedite cases and should not delay cases for their own personal gain or convenience."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A licensed attorney may practice law in a jurisdiction where they are not licensed in an emergency situation if it can avoid a substantial injustice."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney must report any attorney or judge's violation of the Rules if it raises a substantial question as to their honesty, trustworthiness, or fitness as a lawyer."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A subordinate lawyer who follows an order to take an action in violation of the Rules is not subject to discipline if the ethical responsibility is debatable."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney has no duty to follow valid procedural rules and court orders that they reasonably believe are defective or invalid."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Attorneys must not engage in conduct involving dishonesty, fraud, deceit, or misrepresentation, even in their private business or personal life that is unrelated to the practice of law."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is permissible for an attorney to talk to members of the jury before or during a trial, so long as the discussion is not about the trial."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney may be disciplined for failing to prevent ethical violations of other members of their law firm."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may sell certain cases of their law practice and retain others, depending on how valuable they are."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who sells their entire practice may subsequently work as in-house counsel for a business."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "During the sale of a law practice, the seller must obtain client consent before sharing detailed information about a client's case with the potential buyer."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "If a client does not respond within 60 days after receiving notice of the proposed sale of their lawyer's practice, their consent to the transfer of their files is presumed."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "If a client does not respond within 90 days after receiving notice of the proposed sale of their lawyer's practice, their consent to the transfer of their files is presumed."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is a violation of the ABA Model Rules of Professional Conduct for a lawyer to sell only a specific area of their law practice."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who sells an area of their practice must cease accepting any matters in that area, including as counsel or co-counsel, after the sale."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer selling their practice is allowed to share client confidences with the potential buyer without client consent during preliminary negotiations."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is required to render pro bono legal services each year."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer should aspire to render at least 10 hours of pro bono legal services each year."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer should aspire to render at least 25 hours of pro bono legal services each year."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer should aspire to render at least 50 hours of pro bono legal services each year."}], "ideal": "True"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer should aspire to render at least 100 hours of pro bono legal services each year."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer shall not represent anyone in connection with a matter in which the lawyer participated personally and substantially as a judge or other adjudicative officer or law clerk to such a person or as an arbitrator, mediator or other third-party neutral, unless 3 years have passed after the laywer's last point of involvement."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer shall not represent anyone in connection with a matter in which the lawyer participated personally and substantially as a judge or other adjudicative officer or law clerk to such a person or as an arbitrator, mediator or other third-party neutral, unless 5 years have passed after the laywer's last point of involvement."}], "ideal": "False"}
+{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer shall not represent anyone in connection with a matter in which the lawyer participated personally and substantially as a judge or other adjudicative officer or law clerk to such a person or as an arbitrator, mediator or other third-party neutral, unless 10 years have passed after the laywer's last point of involvement."}], "ideal": "False"}

evals/evals/registry/data/actors-sequence/samples.jsonl

@@ -0,0 +1,100 @@
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMALAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Lancelot:0,Arthur:7,Merlin:5+6,Arthur:11,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Merlin:1+2,Arthur:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAMMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Merlin:1+2,Merlin:3+4,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMLLLLLMLLAAMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Lancelot:1,Lancelot:2,Lancelot:3,Lancelot:4,Lancelot:5,Merlin:3+4,Lancelot:6,Lancelot:7,Arthur:13,Arthur:4,Merlin:5+6,Merlin:7+8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMMMMAMMMMMMMALAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Merlin:11+12,Arthur:3,Merlin:13+14,Merlin:15+16,Merlin:17+18,Merlin:19+20,Merlin:21+22,Merlin:23+24,Merlin:25+26,Arthur:8,Lancelot:0,Arthur:8,Merlin:27+28,Arthur:10,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMAMAMAMAMAMAMAMAMAMAMAMAMALLLLMAMAMAMAMAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Merlin:5+6,Arthur:11,Merlin:7+8,Arthur:15,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,Merlin:13+14,Arthur:5,Merlin:15+16,Arthur:7,Merlin:17+18,Arthur:9,Merlin:19+20,Arthur:2,Merlin:21+22,Arthur:4,Merlin:23+24,Arthur:6,Merlin:25+26,Arthur:8,Merlin:27+28,Arthur:10,Lancelot:0,Lancelot:1,Lancelot:2,Lancelot:3,Merlin:29+30,Arthur:3,Merlin:31+32,Arthur:5,Merlin:33+34,Arthur:7,Merlin:35+36,Arthur:9,Merlin:37+38,Arthur:11,Merlin:39+40,Arthur:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLLLLLLAMLLLLLLLLA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Lancelot:3,Lancelot:4,Lancelot:5,Lancelot:6,Lancelot:7,Arthur:13,Merlin:1+2,Lancelot:8,Lancelot:9,Lancelot:10,Lancelot:11,Lancelot:12,Lancelot:13,Lancelot:14,Lancelot:15,Arthur:6,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLALALALALALALALALALAMMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Lancelot:2,Arthur:3,Lancelot:3,Arthur:6,Lancelot:4,Arthur:10,Lancelot:5,Arthur:5,Lancelot:6,Arthur:11,Lancelot:7,Arthur:8,Lancelot:8,Arthur:16,Lancelot:9,Arthur:15,Lancelot:10,Arthur:1,Merlin:1+2,Merlin:3+4,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMMLAALLALLAAALLLAAAALLLA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Merlin:5+6,Lancelot:1,Arthur:7,Arthur:8,Lancelot:2,Lancelot:3,Arthur:5,Lancelot:4,Lancelot:5,Arthur:9,Arthur:14,Arthur:5,Lancelot:6,Lancelot:7,Lancelot:8,Arthur:15,Arthur:6,Arthur:11,Arthur:2,Lancelot:9,Lancelot:10,Lancelot:11,Arthur:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMAAAAAAAAAAAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Arthur:11,Arthur:2,Arthur:3,Arthur:5,Arthur:8,Arthur:13,Arthur:4,Arthur:7,Arthur:11,Arthur:2,Arthur:3,Arthur:5,Arthur:8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAAAAAAAMAAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Arthur:5,Arthur:8,Arthur:13,Arthur:4,Arthur:7,Arthur:11,Merlin:3+4,Arthur:7,Arthur:11,Arthur:2,Arthur:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAMAMMMMAAAAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Merlin:1+2,Arthur:3,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Arthur:1,Arthur:1,Arthur:2,Arthur:3,Arthur:5,Arthur:8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAMMAMALLMMLLMMLLMMLLMMLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Merlin:1+2,Merlin:3+4,Arthur:7,Merlin:5+6,Arthur:11,Lancelot:2,Lancelot:3,Merlin:7+8,Merlin:9+10,Lancelot:4,Lancelot:5,Merlin:11+12,Merlin:13+14,Lancelot:6,Lancelot:7,Merlin:15+16,Merlin:17+18,Lancelot:8,Lancelot:9,Merlin:19+20,Merlin:21+22,Lancelot:10,Lancelot:11,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMLLAAMMLLAAMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Merlin:7+8,Merlin:9+10,Lancelot:2,Lancelot:3,Arthur:5,Arthur:8,Merlin:11+12,Merlin:13+14,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMMMMAAAAAMALAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Merlin:11+12,Arthur:3,Arthur:5,Arthur:8,Arthur:13,Arthur:4,Merlin:13+14,Arthur:5,Lancelot:0,Arthur:5,Merlin:15+16,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMALMAMALMAMAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Lancelot:0,Merlin:5+6,Arthur:11,Merlin:7+8,Arthur:15,Lancelot:1,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,Lancelot:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLLLLLLLLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Lancelot:3,Lancelot:4,Lancelot:5,Lancelot:6,Lancelot:7,Lancelot:8,Lancelot:9,Lancelot:10,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMAAAAAAAAAAAAAAAAAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Arthur:7,Arthur:11,Arthur:2,Arthur:3,Arthur:5,Arthur:8,Arthur:13,Arthur:4,Arthur:7,Arthur:11,Arthur:2,Arthur:3,Arthur:5,Arthur:8,Arthur:13,Arthur:4,Arthur:7,Arthur:11,Arthur:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMMMMMMMMMMMMMMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Merlin:11+12,Merlin:13+14,Merlin:15+16,Merlin:17+18,Merlin:19+20,Merlin:21+22,Merlin:23+24,Merlin:25+26,Merlin:27+28,Merlin:29+30,Merlin:31+32,Merlin:33+34,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMMMAMMMMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Arthur:1,Merlin:11+12,Merlin:13+14,Merlin:15+16,Merlin:17+18,Arthur:9,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMMMMMAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Merlin:11+12,Arthur:3,Merlin:13+14,Arthur:5,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAAAAAAAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Arthur:3,Arthur:5,Arthur:8,Arthur:13,Arthur:4,Merlin:1+2,Arthur:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLLLLLLLLLLLLAMMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Lancelot:3,Lancelot:4,Lancelot:5,Lancelot:6,Lancelot:7,Lancelot:8,Lancelot:9,Lancelot:10,Lancelot:11,Lancelot:12,Lancelot:13,Arthur:4,Merlin:1+2,Merlin:3+4,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAAAMMMAMMLLAAMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Arthur:3,Merlin:1+2,Merlin:3+4,Merlin:5+6,Arthur:11,Merlin:7+8,Merlin:9+10,Lancelot:2,Lancelot:3,Arthur:5,Arthur:8,Merlin:11+12,Merlin:13+14,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMAAMMMAAMAMALAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Arthur:11,Arthur:2,Merlin:7+8,Merlin:9+10,Merlin:11+12,Arthur:3,Arthur:5,Merlin:13+14,Arthur:5,Merlin:15+16,Arthur:7,Lancelot:0,Arthur:7,Merlin:17+18,Arthur:9,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMAMAMAMAMAMAMAMAMAMAMAMAMAMAMAMAMAMAMAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Merlin:5+6,Arthur:11,Merlin:7+8,Arthur:15,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,Merlin:13+14,Arthur:5,Merlin:15+16,Arthur:7,Merlin:17+18,Arthur:9,Merlin:19+20,Arthur:2,Merlin:21+22,Arthur:4,Merlin:23+24,Arthur:6,Merlin:25+26,Arthur:8,Merlin:27+28,Arthur:10,Merlin:29+30,Arthur:3,Merlin:31+32,Arthur:5,Merlin:33+34,Arthur:7,Merlin:35+36,Arthur:9,Merlin:37+38,Arthur:11,Merlin:39+40,Arthur:4,Lancelot:0,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Lancelot:0,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMALLMAMALMAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Lancelot:0,Lancelot:1,Merlin:5+6,Arthur:11,Merlin:7+8,Arthur:15,Lancelot:2,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMALMAMAMAMALMAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Lancelot:0,Merlin:5+6,Arthur:11,Merlin:7+8,Arthur:15,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,Lancelot:1,Merlin:13+14,Arthur:5,Merlin:15+16,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMALAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Lancelot:0,Arthur:7,Merlin:5+6,Arthur:11,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMAMMLALMLAMLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Arthur:7,Merlin:5+6,Merlin:7+8,Lancelot:1,Arthur:9,Lancelot:2,Merlin:9+10,Lancelot:3,Arthur:3,Merlin:11+12,Lancelot:4,Lancelot:5,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAAAAMALMMLLML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Arthur:5,Arthur:8,Arthur:13,Merlin:3+4,Arthur:7,Lancelot:0,Merlin:5+6,Merlin:7+8,Lancelot:1,Lancelot:2,Merlin:9+10,Lancelot:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLMMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Merlin:3+4,Merlin:5+6,Merlin:7+8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLLMLAALAALMLALLMMMLMLLMMMAAML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Lancelot:3,Merlin:1+2,Lancelot:4,Arthur:6,Arthur:10,Lancelot:5,Arthur:5,Arthur:10,Lancelot:6,Merlin:3+4,Lancelot:7,Arthur:11,Lancelot:8,Lancelot:9,Merlin:5+6,Merlin:7+8,Merlin:9+10,Lancelot:10,Merlin:11+12,Lancelot:11,Lancelot:12,Merlin:13+14,Merlin:15+16,Merlin:17+18,Arthur:9,Arthur:17,Merlin:19+20,Lancelot:13,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLMLLLAMLLAMMALLMMMMMAMLLMAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Merlin:1+2,Lancelot:3,Lancelot:4,Lancelot:5,Arthur:9,Merlin:3+4,Lancelot:6,Lancelot:7,Arthur:13,Merlin:5+6,Merlin:7+8,Arthur:15,Lancelot:8,Lancelot:9,Merlin:9+10,Merlin:11+12,Merlin:13+14,Merlin:15+16,Merlin:17+18,Arthur:9,Merlin:19+20,Lancelot:10,Lancelot:11,Merlin:21+22,Arthur:4,Merlin:23+24,Arthur:6,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAALALMLLALMLAMALALLAMLALMLA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Lancelot:2,Arthur:4,Lancelot:3,Merlin:1+2,Lancelot:4,Lancelot:5,Arthur:9,Lancelot:6,Merlin:3+4,Lancelot:7,Arthur:11,Merlin:5+6,Arthur:11,Lancelot:8,Arthur:9,Lancelot:9,Lancelot:10,Arthur:1,Merlin:7+8,Lancelot:11,Arthur:2,Lancelot:12,Merlin:9+10,Lancelot:13,Arthur:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLLAAMLMAMLAAAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Merlin:3+4,Lancelot:2,Merlin:5+6,Arthur:11,Merlin:7+8,Lancelot:3,Arthur:11,Arthur:2,Arthur:3,Merlin:9+10,Arthur:1,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMAMMLMALLMMAMMMLMMMAMLALAAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Arthur:7,Merlin:5+6,Merlin:7+8,Lancelot:1,Merlin:9+10,Arthur:1,Lancelot:2,Lancelot:3,Merlin:11+12,Merlin:13+14,Arthur:5,Merlin:15+16,Merlin:17+18,Merlin:19+20,Lancelot:4,Merlin:21+22,Merlin:23+24,Merlin:25+26,Arthur:8,Merlin:27+28,Lancelot:5,Arthur:13,Lancelot:6,Arthur:9,Arthur:15,Lancelot:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMLAMLLLMMALALLLLMMMALAAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Lancelot:1,Arthur:3,Merlin:3+4,Lancelot:2,Lancelot:3,Lancelot:4,Merlin:5+6,Merlin:7+8,Arthur:15,Lancelot:5,Arthur:10,Lancelot:6,Lancelot:7,Lancelot:8,Lancelot:9,Merlin:9+10,Merlin:11+12,Merlin:13+14,Arthur:5,Lancelot:10,Arthur:1,Arthur:1,Lancelot:11,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMALLLMMLALAAAM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Arthur:3,Lancelot:2,Lancelot:3,Lancelot:4,Merlin:3+4,Merlin:5+6,Lancelot:5,Arthur:11,Lancelot:6,Arthur:7,Arthur:13,Arthur:4,Merlin:7+8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAMMAAAMLLMAMLLMMLLMAM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Merlin:1+2,Merlin:3+4,Arthur:7,Arthur:11,Arthur:2,Merlin:5+6,Lancelot:2,Lancelot:3,Merlin:7+8,Arthur:15,Merlin:9+10,Lancelot:4,Lancelot:5,Merlin:11+12,Merlin:13+14,Lancelot:6,Lancelot:7,Merlin:15+16,Arthur:7,Merlin:17+18,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMALMALLLALALLMLLMMMALMLLMLLMAAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Arthur:3,Lancelot:1,Merlin:3+4,Arthur:7,Lancelot:2,Lancelot:3,Lancelot:4,Arthur:7,Lancelot:5,Arthur:12,Lancelot:6,Lancelot:7,Merlin:5+6,Lancelot:8,Lancelot:9,Merlin:7+8,Merlin:9+10,Merlin:11+12,Arthur:3,Lancelot:10,Merlin:13+14,Lancelot:11,Lancelot:12,Merlin:15+16,Lancelot:13,Lancelot:14,Merlin:17+18,Arthur:9,Arthur:17,Lancelot:15,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMLLAMMMAMAALMMLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Lancelot:0,Lancelot:1,Arthur:1,Merlin:5+6,Merlin:7+8,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,Arthur:5,Lancelot:2,Merlin:13+14,Merlin:15+16,Lancelot:3,Lancelot:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLALAMMMLLMLLMLLLMLMLMAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Arthur:3,Lancelot:3,Arthur:6,Merlin:1+2,Merlin:3+4,Merlin:5+6,Lancelot:4,Lancelot:5,Merlin:7+8,Lancelot:6,Lancelot:7,Merlin:9+10,Lancelot:8,Lancelot:9,Lancelot:10,Merlin:11+12,Lancelot:11,Merlin:13+14,Lancelot:12,Merlin:15+16,Arthur:7,Arthur:13,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMAMMLMMLMLLMMMMAMLLAML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Arthur:7,Merlin:5+6,Merlin:7+8,Lancelot:0,Merlin:9+10,Merlin:11+12,Lancelot:1,Merlin:13+14,Lancelot:2,Lancelot:3,Merlin:15+16,Merlin:17+18,Merlin:19+20,Merlin:21+22,Arthur:4,Merlin:23+24,Lancelot:4,Lancelot:5,Arthur:9,Merlin:25+26,Lancelot:6,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMLLAALLAMALLLLAMMLALLAALMLML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Lancelot:1,Lancelot:2,Arthur:3,Arthur:5,Lancelot:3,Lancelot:4,Arthur:7,Merlin:5+6,Arthur:11,Lancelot:5,Lancelot:6,Lancelot:7,Lancelot:8,Arthur:15,Merlin:7+8,Merlin:9+10,Lancelot:9,Arthur:9,Lancelot:10,Lancelot:11,Arthur:2,Arthur:3,Lancelot:12,Merlin:11+12,Lancelot:13,Merlin:13+14,Lancelot:14,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAAALLLM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Arthur:3,Lancelot:2,Lancelot:3,Lancelot:4,Merlin:1+2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAMLALLAMLAMAALMALLAALAMLLM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Merlin:1+2,Lancelot:2,Arthur:4,Lancelot:3,Lancelot:4,Arthur:7,Merlin:3+4,Lancelot:5,Arthur:9,Merlin:5+6,Arthur:11,Arthur:2,Lancelot:6,Merlin:7+8,Arthur:15,Lancelot:7,Lancelot:8,Arthur:15,Arthur:6,Lancelot:9,Arthur:15,Merlin:9+10,Lancelot:10,Lancelot:11,Merlin:11+12,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAAMMLAAMMMLLMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Merlin:1+2,Merlin:3+4,Lancelot:2,Arthur:6,Arthur:8,Merlin:5+6,Merlin:7+8,Merlin:9+10,Lancelot:3,Lancelot:4,Merlin:11+12,Merlin:13+14,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Arthur:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMLMAMLMLALALMLAMMLMLM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Lancelot:2,Merlin:3+4,Arthur:7,Merlin:5+6,Lancelot:3,Merlin:7+8,Lancelot:4,Arthur:12,Lancelot:5,Arthur:7,Lancelot:6,Merlin:9+10,Lancelot:7,Arthur:7,Merlin:11+12,Merlin:13+14,Lancelot:8,Merlin:15+16,Lancelot:9,Merlin:17+18,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMLMAMMMMLMMAAAMAAALMAAAMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Lancelot:1,Merlin:3+4,Arthur:7,Merlin:5+6,Merlin:7+8,Merlin:9+10,Merlin:11+12,Lancelot:2,Merlin:13+14,Merlin:15+16,Arthur:7,Arthur:13,Arthur:4,Merlin:17+18,Arthur:9,Arthur:17,Arthur:8,Lancelot:3,Merlin:19+20,Arthur:2,Arthur:2,Arthur:4,Merlin:21+22,Merlin:23+24,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMAAAMMLAAMLMAMLLMLALLLALLA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Arthur:3,Arthur:5,Arthur:8,Merlin:3+4,Merlin:5+6,Lancelot:2,Arthur:8,Arthur:10,Merlin:7+8,Lancelot:3,Merlin:9+10,Arthur:1,Merlin:11+12,Lancelot:4,Lancelot:5,Merlin:13+14,Lancelot:6,Arthur:10,Lancelot:7,Lancelot:8,Lancelot:9,Arthur:17,Lancelot:10,Lancelot:11,Arthur:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMALMMLLLMLMMMLAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Arthur:7,Lancelot:0,Merlin:5+6,Merlin:7+8,Lancelot:1,Lancelot:2,Lancelot:3,Merlin:9+10,Lancelot:4,Merlin:11+12,Merlin:13+14,Merlin:15+16,Lancelot:5,Arthur:11,Lancelot:6,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMMAAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Merlin:5+6,Arthur:11,Arthur:2,Lancelot:1,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLLALAMAAALLALMMLLLMLMAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Lancelot:1,Arthur:1,Lancelot:2,Arthur:3,Merlin:3+4,Arthur:7,Arthur:11,Arthur:2,Lancelot:3,Lancelot:4,Arthur:7,Lancelot:5,Merlin:5+6,Merlin:7+8,Lancelot:6,Lancelot:7,Lancelot:8,Merlin:9+10,Lancelot:9,Merlin:11+12,Arthur:3,Arthur:5,Arthur:8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMLAMAAALMMMLALAALAMALLMAAMLMLM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Lancelot:1,Arthur:5,Merlin:5+6,Arthur:11,Arthur:2,Arthur:3,Lancelot:2,Merlin:7+8,Merlin:9+10,Merlin:11+12,Lancelot:3,Arthur:5,Lancelot:4,Arthur:9,Arthur:13,Lancelot:5,Arthur:8,Merlin:13+14,Arthur:5,Lancelot:6,Lancelot:7,Merlin:15+16,Arthur:7,Arthur:13,Merlin:17+18,Lancelot:8,Merlin:19+20,Lancelot:9,Merlin:21+22,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMLLLLLLLAALLLLMMMMALLLAMMAAAAML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Lancelot:1,Lancelot:2,Lancelot:3,Lancelot:4,Lancelot:5,Lancelot:6,Lancelot:7,Arthur:13,Arthur:4,Lancelot:8,Lancelot:9,Lancelot:10,Lancelot:11,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Arthur:1,Lancelot:12,Lancelot:13,Lancelot:14,Arthur:5,Merlin:11+12,Merlin:13+14,Arthur:5,Arthur:9,Arthur:14,Arthur:5,Merlin:15+16,Lancelot:15,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: M. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: M. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMLLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Lancelot:0,Lancelot:1,Lancelot:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLMAALMMALLLMAAAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Merlin:1+2,Arthur:3,Arthur:5,Lancelot:3,Merlin:3+4,Merlin:5+6,Arthur:11,Lancelot:4,Lancelot:5,Lancelot:6,Merlin:7+8,Arthur:15,Arthur:6,Arthur:11,Arthur:2,Arthur:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAAAMLALLM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Arthur:5,Arthur:8,Merlin:3+4,Lancelot:0,Arthur:4,Lancelot:1,Lancelot:2,Merlin:5+6,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMMMMMMMLLLAMLMLMMLMMLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Merlin:11+12,Merlin:13+14,Lancelot:2,Lancelot:3,Lancelot:4,Arthur:7,Merlin:15+16,Lancelot:5,Merlin:17+18,Lancelot:6,Merlin:19+20,Merlin:21+22,Lancelot:7,Merlin:23+24,Merlin:25+26,Lancelot:8,Lancelot:9,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMALLM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Arthur:3,Lancelot:1,Lancelot:2,Merlin:3+4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MALMLLMLAAMMMMMLLMAALMLLMMALA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Lancelot:0,Merlin:3+4,Lancelot:1,Lancelot:2,Merlin:5+6,Lancelot:3,Arthur:9,Arthur:12,Merlin:7+8,Merlin:9+10,Merlin:11+12,Merlin:13+14,Merlin:15+16,Lancelot:4,Lancelot:5,Merlin:17+18,Arthur:9,Arthur:17,Lancelot:6,Merlin:19+20,Lancelot:7,Lancelot:8,Merlin:21+22,Merlin:23+24,Arthur:6,Lancelot:9,Arthur:15,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMLAMAML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Lancelot:0,Arthur:4,Merlin:5+6,Arthur:11,Merlin:7+8,Lancelot:1,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MALLLAALMMMMLML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Lancelot:0,Lancelot:1,Lancelot:2,Arthur:3,Arthur:5,Lancelot:3,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Lancelot:4,Merlin:11+12,Lancelot:5,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLAALLLAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Arthur:2,Arthur:2,Lancelot:1,Lancelot:2,Lancelot:3,Arthur:5,Lancelot:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLAML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Arthur:1,Merlin:1+2,Lancelot:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMAAMALALAMAAAAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Arthur:3,Arthur:5,Merlin:3+4,Arthur:7,Lancelot:2,Arthur:9,Lancelot:3,Arthur:12,Merlin:5+6,Arthur:11,Arthur:2,Arthur:3,Arthur:5,Lancelot:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLLMLLLAMAALLMAMLLMAALM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Lancelot:1,Merlin:3+4,Lancelot:2,Lancelot:3,Lancelot:4,Arthur:7,Merlin:5+6,Arthur:11,Arthur:2,Lancelot:5,Lancelot:6,Merlin:7+8,Arthur:15,Merlin:9+10,Lancelot:7,Lancelot:8,Merlin:11+12,Arthur:3,Arthur:5,Lancelot:9,Merlin:13+14,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLLMLALLAAMMLLALML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Lancelot:3,Merlin:1+2,Lancelot:4,Arthur:6,Lancelot:5,Lancelot:6,Arthur:11,Arthur:2,Merlin:3+4,Merlin:5+6,Lancelot:7,Lancelot:8,Arthur:15,Lancelot:9,Merlin:7+8,Lancelot:10,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMLMLALAMMAMALALAALMMALML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Lancelot:2,Merlin:3+4,Lancelot:3,Arthur:7,Lancelot:4,Arthur:11,Merlin:5+6,Merlin:7+8,Arthur:15,Merlin:9+10,Arthur:1,Lancelot:5,Arthur:6,Lancelot:6,Arthur:12,Arthur:3,Lancelot:7,Merlin:11+12,Merlin:13+14,Arthur:5,Lancelot:8,Merlin:15+16,Lancelot:9,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMLALLLAALMMMALAAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Lancelot:1,Arthur:5,Lancelot:2,Lancelot:3,Lancelot:4,Arthur:7,Arthur:11,Lancelot:5,Merlin:5+6,Merlin:7+8,Merlin:9+10,Arthur:1,Lancelot:6,Arthur:7,Arthur:13,Arthur:4,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLLLMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Lancelot:1,Lancelot:2,Merlin:3+4,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAMAMMALLAAALMMLLLAALMALAMMMLA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Merlin:5+6,Merlin:7+8,Arthur:15,Lancelot:0,Lancelot:1,Arthur:1,Arthur:2,Arthur:3,Lancelot:2,Merlin:9+10,Merlin:11+12,Lancelot:3,Lancelot:4,Lancelot:5,Arthur:9,Arthur:14,Lancelot:6,Merlin:13+14,Arthur:5,Lancelot:7,Arthur:12,Merlin:15+16,Merlin:17+18,Merlin:19+20,Lancelot:8,Arthur:8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMMAMLLMMLALALLAMLALMALLALMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Merlin:3+4,Arthur:7,Merlin:5+6,Lancelot:2,Lancelot:3,Merlin:7+8,Merlin:9+10,Lancelot:4,Arthur:4,Lancelot:5,Arthur:9,Lancelot:6,Lancelot:7,Arthur:13,Merlin:11+12,Lancelot:8,Arthur:10,Lancelot:9,Merlin:13+14,Arthur:5,Lancelot:10,Lancelot:11,Arthur:2,Lancelot:12,Merlin:15+16,Arthur:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLLAAALML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Lancelot:2,Arthur:3,Arthur:5,Arthur:8,Lancelot:3,Merlin:1+2,Lancelot:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMAMLAMAMALMAMAAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Arthur:3,Merlin:3+4,Lancelot:1,Arthur:5,Merlin:5+6,Arthur:11,Merlin:7+8,Arthur:15,Lancelot:2,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,Arthur:5,Arthur:8,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMAMMLLMMAAALAAM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Arthur:7,Merlin:5+6,Merlin:7+8,Lancelot:0,Lancelot:1,Merlin:9+10,Merlin:11+12,Arthur:3,Arthur:5,Arthur:8,Lancelot:2,Arthur:10,Arthur:1,Merlin:13+14,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMLMLMAMAMLMLMLALMLAMMMLMMLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Lancelot:1,Merlin:5+6,Lancelot:2,Merlin:7+8,Arthur:15,Merlin:9+10,Arthur:1,Merlin:11+12,Lancelot:3,Merlin:13+14,Lancelot:4,Merlin:15+16,Lancelot:5,Arthur:11,Lancelot:6,Merlin:17+18,Lancelot:7,Arthur:15,Merlin:19+20,Merlin:21+22,Merlin:23+24,Lancelot:8,Merlin:25+26,Merlin:27+28,Lancelot:9,Lancelot:10,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLLAMALMMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Lancelot:1,Arthur:1,Merlin:3+4,Arthur:7,Lancelot:2,Merlin:5+6,Merlin:7+8,Merlin:9+10,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMLLMMMAMMLML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Lancelot:1,Lancelot:2,Merlin:5+6,Merlin:7+8,Merlin:9+10,Arthur:1,Merlin:11+12,Merlin:13+14,Lancelot:3,Merlin:15+16,Lancelot:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMMMMAAAALLALLMMMMMAMAALLLMMMAM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Arthur:1,Arthur:1,Arthur:2,Arthur:3,Lancelot:0,Lancelot:1,Arthur:1,Lancelot:2,Lancelot:3,Merlin:11+12,Merlin:13+14,Merlin:15+16,Merlin:17+18,Merlin:19+20,Arthur:2,Merlin:21+22,Arthur:4,Arthur:6,Lancelot:4,Lancelot:5,Lancelot:6,Merlin:23+24,Merlin:25+26,Merlin:27+28,Arthur:10,Merlin:29+30,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMAAAMLMMMLLLMMAAAMAAALAMLL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Arthur:3,Arthur:5,Arthur:8,Merlin:3+4,Lancelot:1,Merlin:5+6,Merlin:7+8,Merlin:9+10,Lancelot:2,Lancelot:3,Lancelot:4,Merlin:11+12,Merlin:13+14,Arthur:5,Arthur:9,Arthur:14,Merlin:15+16,Arthur:7,Arthur:13,Arthur:4,Lancelot:5,Arthur:9,Merlin:17+18,Lancelot:6,Lancelot:7,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMLLLALLAAAMMALMLLAMAMAAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Lancelot:1,Lancelot:2,Lancelot:3,Arthur:5,Lancelot:4,Lancelot:5,Arthur:9,Arthur:14,Arthur:5,Merlin:3+4,Merlin:5+6,Arthur:11,Lancelot:6,Merlin:7+8,Lancelot:7,Lancelot:8,Arthur:15,Merlin:9+10,Arthur:1,Merlin:11+12,Arthur:3,Arthur:5,Lancelot:9,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMAAMMMMMMAMMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Arthur:3,Arthur:5,Merlin:3+4,Merlin:5+6,Merlin:7+8,Merlin:9+10,Merlin:11+12,Merlin:13+14,Arthur:5,Merlin:15+16,Merlin:17+18,Arthur:9,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMMMMAMALML. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Merlin:3+4,Merlin:5+6,Merlin:7+8,Arthur:15,Merlin:9+10,Arthur:1,Lancelot:1,Merlin:11+12,Lancelot:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LLMAMAMAMLLAMMAL. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Lancelot:1,Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,Merlin:5+6,Arthur:11,Merlin:7+8,Lancelot:2,Lancelot:3,Arthur:5,Merlin:9+10,Merlin:11+12,Arthur:3,Lancelot:4,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Arthur:2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: M. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MLLLLMAAMMAAALALALAA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Lancelot:0,Lancelot:1,Lancelot:2,Lancelot:3,Merlin:3+4,Arthur:7,Arthur:11,Merlin:5+6,Merlin:7+8,Arthur:15,Arthur:6,Arthur:11,Lancelot:4,Arthur:5,Lancelot:5,Arthur:10,Lancelot:6,Arthur:6,Arthur:12,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MMLMAMLAAMAMMMAALLLAMLALLAMMMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Merlin:3+4,Lancelot:0,Merlin:5+6,Arthur:11,Merlin:7+8,Lancelot:1,Arthur:9,Arthur:10,Merlin:9+10,Arthur:1,Merlin:11+12,Merlin:13+14,Merlin:15+16,Arthur:7,Arthur:13,Lancelot:2,Lancelot:3,Lancelot:4,Arthur:7,Merlin:17+18,Lancelot:5,Arthur:13,Lancelot:6,Lancelot:7,Arthur:13,Merlin:19+20,Merlin:21+22,Merlin:23+24,Arthur:6,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: MAAALMMAAAMALAMALMLLAMM. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Merlin:1+2,Arthur:3,Arthur:5,Arthur:8,Lancelot:0,Merlin:3+4,Merlin:5+6,Arthur:11,Arthur:2,Arthur:3,Merlin:7+8,Arthur:15,Lancelot:1,Arthur:6,Merlin:9+10,Arthur:1,Lancelot:2,Merlin:11+12,Lancelot:3,Lancelot:4,Arthur:7,Merlin:13+14,Merlin:15+16,"}
+{"input": [{"role": "system", "content": "This is an exchange between Merlin, Arthur and Lancelot. Merlin outputs short tests. First output is: '1+2', second is: '3+4', ...  Arthur outputs the sum of last two digits that he had seen. Lancelot outputs an increasing integers number, starting from 0. The sequence in which the actors act: LMAMA. Output that exchange using the following syntax (including the trailing comma): Actor:<output>,..."}], "ideal": "Lancelot:0,Merlin:1+2,Arthur:3,Merlin:3+4,Arthur:7,"}

evals/evals/registry/data/anagrams/fewshot.jsonl

@@ -0,0 +1,5 @@
+{"sample":[{"role":"system","content":"cebyiilsatisc","name":"example_user"},{"role":"system","content":"accessibility","name":"example_assistant"}]}
+{"sample":[{"role":"system","content":"mdoaoiotcnamc","name":"example_user"},{"role":"system","content":"accommodation","name":"example_assistant"}]}
+{"sample":[{"role":"system","content":"tsiooamnacomcd","name":"example_user"},{"role":"system","content":"accommodations","name":"example_assistant"}]}
+{"sample":[{"role":"system","content":"nycaoniapmgc","name":"example_user"},{"role":"system","content":"accompanying","name":"example_assistant"}]}
+{"sample":[{"role":"system","content":"hedamccoslip","name":"example_user"},{"role":"system","content":"accomplished","name":"example_assistant"}]}

evals/evals/registry/data/anagrams/samples.jsonl

@@ -0,0 +1,357 @@
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aotliibcyauntc"}],"ideal":"accountability"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"toetdaicnciar"}],"ideal":"accreditation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"etmsnievahec"}],"ideal":"achievements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eadlwnekdcog"}],"ideal":"acknowledged"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"cnitaiiqouss"}],"ideal":"acquisitions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iolidnltadya"}],"ideal":"additionally"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"mideadetsrni"}],"ideal":"administered"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nidtiamrtsaoin"}],"ideal":"administration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"adniesattmiivr"}],"ideal":"administrative"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"moirastidrant"}],"ideal":"administrator"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aianrrmtstsdio"}],"ideal":"administrators"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iemrentatsved"}],"ideal":"advertisement"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dseamvresettni"}],"ideal":"advertisements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"urgrticalalu"}],"ideal":"agricultural"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lbctalaaihpe"}],"ideal":"alphabetical"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"itlleearvynta"}],"ideal":"alternatively"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aisavteltnre"}],"ideal":"alternatives"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"onmuateecnnn"}],"ideal":"announcement"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"uctmnnneneaos"}],"ideal":"announcements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tryolhapoong"}],"ideal":"anthropology"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aoniscipaplt"}],"ideal":"applications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"apmetintopns"}],"ideal":"appointments"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"apptaioecinr"}],"ideal":"appreciation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rpsprnoiaatpio"}],"ideal":"appropriations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ptapimeyoaxlr"}],"ideal":"approximately"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"riclauehtacrt"}],"ideal":"architectural"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"etcucrrtiaeh"}],"ideal":"architecture"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nrmasantrege"}],"ideal":"arrangements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oossicasnita"}],"ideal":"associations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ohtinitnatcuea"}],"ideal":"authentication"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"hotnzaatoiiur"}],"ideal":"authorization"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ilclyaoamttau"}],"ideal":"automatically"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"vtiiillyabaa"}],"ideal":"availability"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"boihirbcaplgi"}],"ideal":"bibliographic"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lrabobiphyig"}],"ideal":"bibliography"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"idoiteivsryb"}],"ideal":"biodiversity"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nooycheotilgb"}],"ideal":"biotechnology"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"anotabdigscr"}],"ideal":"broadcasting"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nauillstcaoc"}],"ideal":"calculations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iatnenclocla"}],"ideal":"cancellation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sepaciilbtia"}],"ideal":"capabilities"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"urldoccaasariv"}],"ideal":"cardiovascular"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"setiefacitcr"}],"ideal":"certificates"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tniirocaiftce"}],"ideal":"certification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"omsipicphnah"}],"ideal":"championship"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iphnhscomaips"}],"ideal":"championships"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rrcticitaescah"}],"ideal":"characteristic"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aetcshatirrcics"}],"ideal":"characteristics"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aiictcazehroatrn"}],"ideal":"characterization"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"zecrtdaeicahr"}],"ideal":"characterized"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"inahcyttisir"}],"ideal":"christianity"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sscenicrutcma"}],"ideal":"circumstances"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iioiilzctavn"}],"ideal":"civilization"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lnaisfctscaioi"}],"ideal":"classification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ablltcirooano"}],"ideal":"collaboration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"cvtblaoielaro"}],"ideal":"collaborative"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"esaltbccleol"}],"ideal":"collectables"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"loicleebtlcs"}],"ideal":"collectibles"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"osmiabtnncoi"}],"ideal":"combinations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nscrismieomo"}],"ideal":"commissioner"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"mnrioosmiecss"}],"ideal":"commissioners"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"hnwceamtolmo"}],"ideal":"commonwealth"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nucmcmaniotoi"}],"ideal":"communication"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sncmnautomciio"}],"ideal":"communications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tpmticbiaioyl"}],"ideal":"compatibility"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"acsnotiemonp"}],"ideal":"compensation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nisttecimoop"}],"ideal":"competitions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pinsomtaoclci"}],"ideal":"complications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pmlynaremitco"}],"ideal":"complimentary"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"enpmshrovieec"}],"ideal":"comprehensive"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aomalctionput"}],"ideal":"computational"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"cotnnecainrto"}],"ideal":"concentration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eotnaincntsroc"}],"ideal":"concentrations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"toiniigcodnn"}],"ideal":"conditioning"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"efciorgnnnce"}],"ideal":"conferencing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lnaoecdfitin"}],"ideal":"confidential"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eaidiniyflntotc"}],"ideal":"confidentiality"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oroitfnanugci"}],"ideal":"configuration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nrcoatimofin"}],"ideal":"confirmation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aglroiocnntstua"}],"ideal":"congratulations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oniloacressgn"}],"ideal":"congressional"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"yvcteiitnnoc"}],"ideal":"connectivity"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ncssnusoeiosc"}],"ideal":"consciousness"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"euqccoeesnns"}],"ideal":"consequences"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nsluenteocqy"}],"ideal":"consequently"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"intacreosnov"}],"ideal":"conservation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"evsaortnceiv"}],"ideal":"conservative"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eocindebaslr"}],"ideal":"considerable"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oirdcainnteso"}],"ideal":"consideration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oaoneiidnstcrs"}],"ideal":"considerations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"yotcntneisls"}],"ideal":"consistently"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oleitncdoasd"}],"ideal":"consolidated"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tdcsnonilaioo"}],"ideal":"consolidation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tinsnoitucto"}],"ideal":"constitution"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"luttnicaoitsno"}],"ideal":"constitutional"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tooncctnruis"}],"ideal":"construction"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"snlionoutact"}],"ideal":"consultation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nmtacnotiaoin"}],"ideal":"contamination"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rortocynpame"}],"ideal":"contemporary"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iucoosluyntn"}],"ideal":"continuously"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"itrntbuoincg"}],"ideal":"contributing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rbunnttiiooc"}],"ideal":"contribution"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"otosrntiibcnu"}],"ideal":"contributions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"trosribuncto"}],"ideal":"contributors"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oeorvrialncst"}],"ideal":"controversial"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"annooncltive"}],"ideal":"conventional"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ercniatovnos"}],"ideal":"conversation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ieaorcnossnvt"}],"ideal":"conversations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"toidncoinroa"}],"ideal":"coordination"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"portcooirnsa"}],"ideal":"corporations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"edseroocperncn"}],"ideal":"correspondence"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rrspogndnoiec"}],"ideal":"corresponding"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tnesroatddme"}],"ideal":"demonstrated"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nsresomdteat"}],"ideal":"demonstrates"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"noenidtsrtmao"}],"ideal":"demonstration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"enealdmtpatr"}],"ideal":"departmental"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"psiidcoenrst"}],"ideal":"descriptions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"otniidetsasn"}],"ideal":"destinations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"enamttidieorn"}],"ideal":"determination"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"teovnpemellad"}],"ideal":"developmental"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eemespdlnvto"}],"ideal":"developments"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"isireoiadnct"}],"ideal":"dictionaries"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rieefnaftidl"}],"ideal":"differential"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"scfteiidulif"}],"ideal":"difficulties"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tsisdibiaiel"}],"ideal":"disabilities"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tsnaedoippdi"}],"ideal":"disappointed"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"cripidainysl"}],"ideal":"disciplinary"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"airdonitsmicin"}],"ideal":"discrimination"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"didhiiesngtsu"}],"ideal":"distinguished"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iiisbturndto"}],"ideal":"distribution"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iobitdrssniut"}],"ideal":"distributions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"usitrobtsird"}],"ideal":"distributors"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ennmotacditou"}],"ideal":"documentation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"onaceemnteuxdoctdteert"}],"ideal":"documentcreatetextnode"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"bdowdlnleaao"}],"ideal":"downloadable"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"trcmilldayaa"}],"ideal":"dramatically"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tcfseveeenisf"}],"ideal":"effectiveness"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ciodlyecenap"}],"ideal":"encyclopedia"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"mcannsehetne"}],"ideal":"enhancements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"gnetnrniieta"}],"ideal":"entertaining"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eittrtannemen"}],"ideal":"entertainment"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rrrepueneent"}],"ideal":"entrepreneur"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"epurenretrens"}],"ideal":"entrepreneurs"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rimvolnentean"}],"ideal":"environmental"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tnesvnnrmioe"}],"ideal":"environments"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"liissathgbne"}],"ideal":"establishing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"isaemlhtntebs"}],"ideal":"establishment"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"snaoxmaienit"}],"ideal":"examinations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"estteoapicxn"}],"ideal":"expectations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pexnsitueder"}],"ideal":"expenditures"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rgieenxepicn"}],"ideal":"experiencing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eiartxeplmen"}],"ideal":"experimental"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nxrrtiaeayrdo"}],"ideal":"extraordinary"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"trelidsnfcai"}],"ideal":"findarticles"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"utfoilictnnya"}],"ideal":"functionality"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tnmsafeualdn"}],"ideal":"fundamentals"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"paciaehglgro"}],"ideal":"geographical"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"garlnteemnvo"}],"ideal":"governmental"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tsaarhdequre"}],"ideal":"headquarters"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ataunnrmhiai"}],"ideal":"humanitarian"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pheioyhacttl"}],"ideal":"hypothetical"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ifndectioitnia"}],"ideal":"identification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aioruisnttll"}],"ideal":"illustration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lostitilruans"}],"ideal":"illustrations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"mteinoetmlpani"}],"ideal":"implementation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"gepmintnimel"}],"ideal":"implementing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aiinmciltosp"}],"ideal":"implications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"petesrvmnoim"}],"ideal":"improvements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aiptoeaiprrnp"}],"ideal":"inappropriate"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rpooaedtricn"}],"ideal":"incorporated"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"inrniaglecys"}],"ideal":"increasingly"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dennneeipdec"}],"ideal":"independence"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"etedpnynlnedi"}],"ideal":"independently"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aiidinnsaplo"}],"ideal":"indianapolis"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"naldviyiliud"}],"ideal":"individually"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"irmaftinolano"}],"ideal":"informational"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rnratsciufurte"}],"ideal":"infrastructure"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"itnoltalsani"}],"ideal":"installation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"laatnnsltisio"}],"ideal":"installations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"siutoinaittnl"}],"ideal":"institutional"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"innsutisoitt"}],"ideal":"institutions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tunaotrslciin"}],"ideal":"instructional"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"isicnrtnotus"}],"ideal":"instructions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lttiumnrnsea"}],"ideal":"instrumental"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rneoattniuismtn"}],"ideal":"instrumentation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tetinaeclllu"}],"ideal":"intellectual"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"leitnegcleni"}],"ideal":"intelligence"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iescatrtonin"}],"ideal":"interactions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"etecrenirfen"}],"ideal":"interference"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tieaenemdtri"}],"ideal":"intermediate"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"arnitoenlitna"}],"ideal":"international"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"anoletlnryitina"}],"ideal":"internationally"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"peetnirntorati"}],"ideal":"interpretation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tronineectsi"}],"ideal":"intersection"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nnvieientotr"}],"ideal":"intervention"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tneeontirnvsi"}],"ideal":"interventions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"irncotinodut"}],"ideal":"introduction"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rtitynocduor"}],"ideal":"introductory"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dttgsieaeivn"}],"ideal":"investigated"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"igaiitentnsov"}],"ideal":"investigation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tnvetisiisnago"}],"ideal":"investigations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"vngtotsiiare"}],"ideal":"investigator"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sgosrttineaiv"}],"ideal":"investigators"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oviklajsceln"}],"ideal":"jacksonville"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nsrctiiiudoj"}],"ideal":"jurisdiction"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"esmktrogdenolw"}],"ideal":"knowledgestorm"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"roaeabilrots"}],"ideal":"laboratories"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eclinietehnst"}],"ideal":"liechtenstein"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"anaeufmcurtd"}],"ideal":"manufactured"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"earutframncu"}],"ideal":"manufacturer"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rtruaamcfusen"}],"ideal":"manufacturers"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"unitncframuag"}],"ideal":"manufacturing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"camshstsestau"}],"ideal":"massachusetts"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aguisnbtmrat"}],"ideal":"masturbating"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tinaamsrtbou"}],"ideal":"masturbation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ilttamamehca"}],"ideal":"mathematical"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"retmsemeunas"}],"ideal":"measurements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eredimranetna"}],"ideal":"mediterranean"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ptorimtolane"}],"ideal":"metropolitan"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"meleulnocsais"}],"ideal":"miscellaneous"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"infiacmoidto"}],"ideal":"modification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tdiasfcominio"}],"ideal":"modifications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"mayiictlinpu"}],"ideal":"municipality"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"totinaioensg"}],"ideal":"negotiations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dgohoohirben"}],"ideal":"neighborhood"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ersevtlehsne"}],"ideal":"nevertheless"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dwlandneounf"}],"ideal":"newfoundland"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"oifnittiaocn"}],"ideal":"notification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"cofiastoinnit"}],"ideal":"notifications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nbetaosvrsoi"}],"ideal":"observations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ailcalyocosn"}],"ideal":"occasionally"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tlaccuioanop"}],"ideal":"occupational"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nposupeirtoit"}],"ideal":"opportunities"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ttmzoipioani"}],"ideal":"optimization"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rnotonsagiia"}],"ideal":"organisation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nsoiaogirntsa"}],"ideal":"organisations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"otaingirnoaz"}],"ideal":"organization"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"zgoaiitorannal"}],"ideal":"organizational"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nrianoosigatz"}],"ideal":"organizations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aamnelytrrpai"}],"ideal":"parliamentary"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aspaitrnctpi"}],"ideal":"participants"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ciadppaertit"}],"ideal":"participated"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"gnittappiiarc"}],"ideal":"participating"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nrtactppoiiia"}],"ideal":"participation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tyurrclaalpi"}],"ideal":"particularly"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"prnesapsrith"}],"ideal":"partnerships"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"enivpaalnnys"}],"ideal":"pennsylvania"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ocfpnerrsaem"}],"ideal":"performances"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lrcpdoyialie"}],"ideal":"periodically"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"easodzprlein"}],"ideal":"personalized"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tcseiveesprp"}],"ideal":"perspectives"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aiprtmlaceahcu"}],"ideal":"pharmaceutical"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"acmalacihsetrpu"}],"ideal":"pharmaceuticals"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lmyoaaorhpcg"}],"ideal":"pharmacology"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dlhaaihleipp"}],"ideal":"philadelphia"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"hraotegphopr"}],"ideal":"photographer"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"grthsroahpoep"}],"ideal":"photographers"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ctooghprapih"}],"ideal":"photographic"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pssieiilstoib"}],"ideal":"possibilities"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"toeatipricrn"}],"ideal":"practitioner"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ctsiroeirnpat"}],"ideal":"practitioners"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pitipaeotnirc"}],"ideal":"precipitation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"qiuiepretrse"}],"ideal":"prerequisite"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rppirocintes"}],"ideal":"prescription"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eaptisternno"}],"ideal":"presentation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"naosiprsetnte"}],"ideal":"presentations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"psarviroeent"}],"ideal":"preservation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eiantdplseri"}],"ideal":"presidential"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"uttiioycpvdr"}],"ideal":"productivity"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sspaonfolire"}],"ideal":"professional"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sosesialonprf"}],"ideal":"professionals"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aichcgpolsyol"}],"ideal":"psychological"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tplabcinsoui"}],"ideal":"publications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"olqancfiiuita"}],"ideal":"qualification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tanifaliuqicso"}],"ideal":"qualifications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"uvqtatnteiai"}],"ideal":"quantitative"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eqinrsiutenao"}],"ideal":"questionnaire"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"imennemdaocrot"}],"ideal":"recommendation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"memsnnecriatdoo"}],"ideal":"recommendations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nocunoeistrctr"}],"ideal":"reconstruction"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aearneitrocl"}],"ideal":"recreational"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"errtoafrrgei"}],"ideal":"refrigerator"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ortiasitgren"}],"ideal":"registration"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ileahritbation"}],"ideal":"rehabilitation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"perlointiash"}],"ideal":"relationship"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"etpolshnisria"}],"ideal":"relationships"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"errenopintaset"}],"ideal":"representation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"etnipnotsrsreea"}],"ideal":"representations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eenvieerptrats"}],"ideal":"representative"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"virnreetesesatp"}],"ideal":"representatives"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eipeetnrgsnr"}],"ideal":"representing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"edrpiuonrtoc"}],"ideal":"reproduction"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rrcuvetidoep"}],"ideal":"reproductive"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nrqeitsreemu"}],"ideal":"requirements"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"saesrtrvneoi"}],"ideal":"reservations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rcypeselivet"}],"ideal":"respectively"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"bsiinlsseroeiitp"}],"ideal":"responsibilities"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eoirntybssilip"}],"ideal":"responsibility"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"cnoitsrsteir"}],"ideal":"restrictions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"uieutsgtrcrnr"}],"ideal":"restructuring"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"yneoloruitrav"}],"ideal":"revolutionary"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tcsaekawahns"}],"ideal":"saskatchewan"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sticaaoitfsn"}],"ideal":"satisfaction"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tossitfcaary"}],"ideal":"satisfactory"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pilhcssrhaos"}],"ideal":"scholarships"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"vrrseseasenc"}],"ideal":"screensavers"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"onrtodcueimcs"}],"ideal":"semiconductor"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"lrhersahsdeo"}],"ideal":"shareholders"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sanifiegincc"}],"ideal":"significance"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nnlayfsgiiitc"}],"ideal":"significantly"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ulyeinmsaultos"}],"ideal":"simultaneously"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"tispoedhsicta"}],"ideal":"sophisticated"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"izgelcispian"}],"ideal":"specializing"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"feyicsillpca"}],"ideal":"specifically"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"cinipifstceao"}],"ideal":"specification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"naecsioiifscpt"}],"ideal":"specifications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"uliiytatsipr"}],"ideal":"spirituality"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rtelasokdhse"}],"ideal":"stakeholders"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sahnsrremttca"}],"ideal":"starsmerchant"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ntgnitrnegesh"}],"ideal":"strengthening"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"coimuebttesm"}],"ideal":"subcommittee"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dicyrrobmuleitse"}],"ideal":"sublimedirectory"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ctsoursbipni"}],"ideal":"subscription"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sbcntpiosiusr"}],"ideal":"subscriptions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"teeuusynlbsq"}],"ideal":"subsequently"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sruessidibia"}],"ideal":"subsidiaries"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"naltaytblusis"}],"ideal":"substantially"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"syluulfcssce"}],"ideal":"successfully"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"yfntlifiuces"}],"ideal":"sufficiently"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"inrdtestpuenne"}],"ideal":"superintendent"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"antsplpueeml"}],"ideal":"supplemental"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"clelvrnuaies"}],"ideal":"surveillance"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iyitstsbinaual"}],"ideal":"sustainability"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nchliaelgocto"}],"ideal":"technological"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ehsetoogcnli"}],"ideal":"technologies"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"eiectlrbhpuc"}],"ideal":"techrepublic"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nolecmisaeinutcomt"}],"ideal":"telecommunications"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"earuermetpst"}],"ideal":"temperatures"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"insoattislme"}],"ideal":"testimonials"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"aknsnvgihgit"}],"ideal":"thanksgiving"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"asrnctasinot"}],"ideal":"transactions"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"itpciraonnrst"}],"ideal":"transcription"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"asrelxnatesu"}],"ideal":"transexuales"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nmatnfatrisoor"}],"ideal":"transformation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iaslstntonar"}],"ideal":"translations"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ssniiotarnms"}],"ideal":"transmission"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rtasepaycnrn"}],"ideal":"transparency"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"sranniottotarp"}],"ideal":"transportation"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"gtotoinhursloeb"}],"ideal":"troubleshooting"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"zaduotrneihu"}],"ideal":"unauthorized"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"dearderguatun"}],"ideal":"undergraduate"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"nsddneitngaru"}],"ideal":"understanding"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"pumntoenemyl"}],"ideal":"unemployment"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"ufeltoanrnuty"}],"ideal":"unfortunately"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"rsietivuesin"}],"ideal":"universities"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"iterncvafioi"}],"ideal":"verification"}
+{"input":[{"role":"system","content":"Find the anagram of the input word (output is a single word)"},{"role":"user","content":"yuvtrlablneii"}],"ideal":"vulnerability"}

evals/evals/registry/data/balance_chemical_equation/samples.jsonl

@@ -0,0 +1,100 @@
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe + Cl2 = FeCl3"}], "ideal": ["2Fe + 3Cl_2 = 2FeCl_3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "KMnO4 + HCl = KCl + MnCl2 + H2O + Cl2"}], "ideal": ["2KMnO4 + 16HCl = 2KCl + 2MnCl2 + 8H2O + 5Cl2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "K4Fe(CN)6 + H2SO4 + H2O = K2SO4 + FeSO4 + (NH4)2SO4 + CO"}], "ideal": ["K4Fe(CN)6 + 6H2SO4 + 6H2O = 2K2SO4 + FeSO4 + 3(NH4)2SO4 + 6CO"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "K4Fe(CN)6 + KMnO4 + H2SO4 = KHSO4 + Fe2(SO4)3 + MnSO4 + HNO3 + CO2 + H2O"}], "ideal": ["10K4Fe(CN)6 + 122KMnO4 + 299H2SO4 = 162KHSO4 + 5Fe2(SO4)3 + 122MnSO4 + 60HNO3 + 60CO2 + 188H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "CuSO4*5H2O = CuSO4 + H2O"}], "ideal": ["CuSO4*5H2O = CuSO4 + 5H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Na + O = Na2O"}], "ideal": ["4Na + O = 2Na2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Na + Cl2 = NaCl"}], "ideal": ["2Na + Cl2 = 2NaCl"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Na + H2O = NaOH + H2"}], "ideal": ["2Na + 2H2O = 2NaOH + H2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Na + H2O + CuSO4 = Cu(OH)2 + Na2SO4 + H2"}], "ideal": ["2Na + 2H2O + CuSO4 = Cu(OH)2 + Na2SO4 + H2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe + CuSO4 = FeSO4 + Cu"}], "ideal": ["Fe + CuSO4 = FeSO4 + Cu"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Mg+ HCI = MgCI2 + H2"}], "ideal": ["Mg+ 2HCI = MgCI2 + H2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "A + O2 = A12O3"}], "ideal": ["4A + 3O2 = 2A12O3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C+ Fe2O3 = Fe + CO2"}], "ideal": ["3C+ 2Fe2O3 = 4Fe + 3CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe + O2 = Fe3O4"}], "ideal": ["3Fe + 2O2 = Fe3O4"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Cu + O2 = CuO"}], "ideal": ["2Cu + O2 = 2CuO"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "2H2 + O2 = 2H2O"}], "ideal": ["H2 + O2 = H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "P + O2 = P2O5"}], "ideal": ["4P + 5O2 = 2P2O5"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C2H5OH + O2 = CO2 + H2O"}], "ideal": ["C2H5OH + 3O2 = 2CO2 + 3H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C+ Fe3O4 = Fe + CO2"}], "ideal": ["2C+ Fe3O4 = 3Fe + 2CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "CO+ Fe2O3 = Fe + CO2"}], "ideal": ["3CO+ Fe2O3 = 2Fe + 3CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "CO+ Fe3O4 = Fe + CO2"}], "ideal": ["4CO+ Fe3O4 = 3Fe + 4CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "A + H2SO4 = AI2(SO4)3 + H2"}], "ideal": ["2A + 3H2SO4 = AI2(SO4)3 + 3H2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "AI + HCI = AICI3 + H2"}], "ideal": ["2AI + 6HCI = 2AICI3 + 3H2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe2O3 + H2SO4 = Fe2(SO4)3 + H2O"}], "ideal": ["Fe2O3 + 3H2SO4 = Fe2(SO4)3 + 3H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NaOH + SO3 = Na2SO4 + H2O"}], "ideal": ["2NaOH + SO3 = Na2SO4 + H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "HCI + Cu(OH)2 = CuCI2 + H2O"}], "ideal": ["2HCI + Cu(OH)2 = CuCI2 + 2H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "HCI + Fe(OH)3 = FeCI3 + H2O"}], "ideal": ["3HCI + Fe(OH)3 = FeCI3 + 3H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "HCI + Al(OH)3 = AICI3 + H2C"}], "ideal": ["3HCI + Al(OH)3 = AICI3 + 3H2C"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "H2SO4 + Cu(OH)2 = CuSO4 + H2O"}], "ideal": ["H2SO4 + Cu(OH)2 = CuSO4 + 2H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "H2SO4 + Fe(OH)3 = Fe2(SO4)3 + H2O"}], "ideal": ["3H2SO4 + 2Fe(OH)3 = Fe2(SO4)3 + 6H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Na2CO3 + HCI = NaCI + H2O + CO2"}], "ideal": ["Na2CO3 + 2HCI = 2NaCI + H2O + CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "KMnO4 = K2MnO4 + MnO2 + O2"}], "ideal": ["2KMnO4 = K2MnO4 + MnO2 + O2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C + Fe2O3 = Fe + CO2"}], "ideal": ["3C + 2Fe2O3 = 4Fe + 3CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "4CO + Fe3O4 = 3Fe + 4CO2"}], "ideal": ["CO + Fe3O4 = Fe + CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al + H2SO4 = Al2(SO4)3 + H2"}], "ideal": ["2Al + 3H2SO4 = Al2(SO4)3 + 3H2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al + HCl = AlCl3 + H2"}], "ideal": ["2Al + 6HCl = 2AlCl3 + 3H2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe2O3 + HCl = FeCl3 + H2O"}], "ideal": ["Fe2O3 + 6HCl = 2FeCl3 + 3H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NaOH + FeCl3 = Fe(OH)3 + NaCl"}], "ideal": ["3NaOH + FeCl3 = Fe(OH)3 + 3NaCl"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Na2CO3 + HCl = NaCl + H2O + CO2"}], "ideal": ["Na2CO3 + 2HCl = 2NaCl + H2O + CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al(OH)3 + H2SO4 = Al2(SO4)3 + H2O"}], "ideal": ["2Al(OH)3 + 3H2SO4 = Al2(SO4)3 + 3H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NaOH + H3PO4 = H2O + Na3PO4"}], "ideal": ["3NaOH + H3PO4 = 3H2O + Na3PO4"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "FeS2 + O2 = Fe2O3 + SO2"}], "ideal": ["4FeS2 + 11O2 = 2Fe2O3 + 8SO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al2O3 = Al + O2"}], "ideal": ["2Al2O3 = 4Al + 3O2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NaHCO3 = Na2CO3 + CO2 + H2O"}], "ideal": ["2NaHCO3 = Na2CO3 + CO2 + H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NH3 + O2 = NO + H2O"}], "ideal": ["4NH3 + 5O2 = 4NO + 6H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NO + O2 = NO2"}], "ideal": ["2NO + O2 = 2NO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NO2 + H2O = HNO3 + NO"}], "ideal": ["3NO2 + H2O = 2HNO3 + NO"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "SiO2 + C = CO + Si"}], "ideal": ["SiO2 + 2C = 2CO + Si"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Mg + N2 = Mg3N2"}], "ideal": ["3Mg + N2 = Mg3N2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Mg + CO2 = MgO + C"}], "ideal": ["2Mg + CO2 = 2MgO + C"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al + Fe2O3 = Al2O3 + Fe"}], "ideal": ["2Al + Fe2O3 = Al2O3 + 2Fe"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al + Co3O4 = Co + Al2O3"}], "ideal": ["8Al + 3Co3O4 = 9Co + 4Al2O3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al + V2O5 = V + Al2O3"}], "ideal": ["10Al + 3V2O5 = 6V + 5Al2O3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al + NaOH + H2O = NaAlO2 + H2 + H2O"}], "ideal": ["2Al + 2NaOH + 6H2O = 2NaAlO2 + 3H2 + 4H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "S + NaOH = Na2S+ Na2SO3 + H2O"}], "ideal": ["3S + 5NaOH = 2Na2S+ Na2SO3 + 3H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "P4 + P2I4 + H2O = PH4I + H3PO4"}], "ideal": ["13P4 + 10P2I4 + 128H2O = 40PH4I + 32H3PO4"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "K4Fe(CN)6 + KMnO4 + H2SO4 = CO2 + KNO3 + H2O + K2SO4 + MnSO4 + Fe2(SO4)3"}], "ideal": ["10K4Fe(CN)6 + 122KMnO4 + 188H2SO4 = 60CO2 + 60KNO3 + 188H2O + 51K2SO4 + 122MnSO4 + 5Fe2(SO4)3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe36Si5 + H3PO4 + K2Cr2O7 = FePO4 + SiO2 + K3PO4 + CrPO4 + H2O"}], "ideal": ["9Fe36Si5 + 994H3PO4 + 231K2Cr2O7 = 378FePO4 + 63SiO2 + 154K3PO4 + 462CrPO4 + 1491H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "K4[Fe(SCN)6] + K2Cr2O7 + H2SO4 = Fe2(SO4)3 + Cr2(SO4)3 + CO2 + H2O + K2SO4 + KNO3"}], "ideal": ["6K4[Fe(SCN)6] + 97K2Cr2O7 + 355 H2SO4 = 3Fe2(SO4)3 + 97Cr2(SO4)3 + 36CO2 + 355H2O + 91K2SO4 + 36KNO3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "P4 + CuSO4 + H2O = Cu3P + H3PO4 + H2SO4"}], "ideal": ["11P4 + 60CuSO4 + 96H2O = 20Cu3P + 24H3PO4 + 60H2SO4"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "2Ca(OH)2 + 3NaHCO3 = 2CaCO3 + NaOH + Na2CO3 + 3H2O"}], "ideal": ["Ca(OH)2 + NaHCO3 = CaCO3 + NaOH + Na2CO3 + H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe42Si7 + H3PO4 + K2Cr2O7 = FePO4 + SiO2 + K3PO4 + CrPO4 + H2O"}], "ideal": ["9Fe42Si7 + 994H3PO4 + 231K2Cr2O7 = 378FePO4 + 63SiO2 + 154K3PO4 + 462CrPO4 + 1491H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "H2 + Ca(CN)2 + NaAlF4 + FeSO4 + MgSiO3 + KI + H3PO4 + PbCrO4 + BrCl + CF2Cl2 + SO2 = PbBr2 + CrCl3 + MgCO3 + K[Al(OH)4] + Fe(SCN)3 + PI3 + Na2SiO3 + CaF2 + H2O"}], "ideal": ["88H2 + 15Ca(CN)2 + 6NaAlF4 + 10FeSO4 + 3MgSiO3 + 6KI + 2H3PO4 + 6PbCrO4 + 12BrCl + 3CF2Cl2 + 20SO2 = 6PbBr2 + 6CrCl3 + 3MgCO3 + 6K[Al(OH)4] + 10Fe(SCN)3 + 2PI3 + 3Na2SiO3 + 15CaF2 + 79H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Fe3O4 + HNO3 = Fe(NO3)3 + NO + H2O"}], "ideal": ["3Fe3O4 + 28HNO3 = 9Fe(NO3)3 + NO + 14H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "P2I4 + H2O + P4 = PH4I + H3PO4"}], "ideal": ["10P2I4 + 128H2O + 13P4 = 40PH4I + 32H3PO4"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "[Cr(N2H4CO)6]4[Cr(CN)6]3 + KMnO4+H2SO4 -> K2Cr2O7 + MnSO4 + CO2 + KNO3 + K2SO4 + H2O"}], "ideal": ["10[Cr(N2H4CO)6]4[Cr(CN)6]3 + 1176KMnO4 + 1399H2SO4 = 35K2Cr2O7 + 1176MnSO4 + 420CO2 + 660KNO3 + 223K2SO4 + 1879H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "BiCl3 + H2O = BiOCl + HCl"}], "ideal": ["BiCl3 + H2O = BiOCl + 2HCl"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "SOCl2 + e = Cl + S + SO2"}], "ideal": ["2SOCl2 + 4e = 4Cl + S + SO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "KClO3 + H2SO4 = KClO4 + ClO2 + K2SO4 + H2O"}], "ideal": ["3KClO3 + H2SO4 = KClO4 + 2ClO2 + K2SO4 + H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Mg3Si2O5(OH)4 + CO2 = MgCO3 + SiO2 + H2O"}], "ideal": ["Mg3Si2O5(OH)4 + 3CO2 = 3MgCO3 + 2SiO2 + 2H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "AsCl3 + NaBH4 = AsH3 + NaCl + BCl3"}], "ideal": ["4AsCl3 + 3NaBH4 = 4AsH3 + 3NaCl + 3BCl3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "CuS + HNO3 = CuSO4 + NO + H2O"}], "ideal": ["3CuS + 8HNO3 = 3CuSO4 + 8NO + H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C4H10 + F2 = C4F10 + HF"}], "ideal": ["C4H10 + 10F2 = C4F10 + 10HF"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NH3 + O2 = N2O + H2O"}], "ideal": ["2NH3 + 2O2 = N2O + 3H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Mg(NO3)2 = MgO + NO2 + O2"}], "ideal": ["2Mg(NO3)2 = 2MgO(s) + 4NO2 + O2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "B2O3 + C + Cl2 = BCl3 + CO"}], "ideal": ["B2O3 + 3C + 3Cl2 = 2BCl3 + 3CO"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "CH3CH2CH2CH3 + O2 = C2H2(CO)2O + H2O"}], "ideal": ["2CH3CH2CH2CH3 + 7O2 = 2C2H2(CO)2O + 8H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "TiCl4 + Na = NaCl + Ti"}], "ideal": ["TiCl4 + 4Na = 4NaCl + Ti"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Cu + HNO3 = Cu(NO3)2 + NO2 + H2O"}], "ideal": ["Cu + 4HNO3 = Cu(NO3)2 + 2NO2 + 2H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "I2 + HNO3 = HIO3 + NO2 + H2O"}], "ideal": ["I2 + 10HNO3 = 2HIO3 + 10NO2 + 4H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Ca5F(PO4)3 + SiO2 + C = CaSiO3 + CaF2 + CO + P"}], "ideal": ["2Ca5F(PO4)3 + 9SiO2 + 15C = 9CaSiO3 + CaF2 + 15CO + 6P"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "GaF3 + Na3PO4 = GaPO4 + NaF"}], "ideal": ["GaF3 + Na3PO4 = GaPO4 + 3NaF"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "NH4F + AlCl3 = NH4Cl + AlF3"}], "ideal": ["3NH4F + AlCl3 = 3NH4Cl + AlF3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Na2CO3 + H3PO4 = Na3PO4 + H2O + CO2"}], "ideal": ["3Na2CO3 + 2H3PO4 = 2Na3PO4 + 3H2O + 3CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "H2SO4 + NaOH = Na2SO4 + H2O"}], "ideal": ["H2SO4 + 2NaOH = Na2SO4 + 2H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Sn + P4 = Sn3P4"}], "ideal": ["3Sn + P4 = Sn3P4"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C4H8O + O2 = CO2 + H2O"}], "ideal": ["2C4H8O + 11O2 = 8CO2 + 8H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Ni + H2SO4 = H2 + Ni2(SO4)3"}], "ideal": ["2Ni + 3H2SO4 = 3H2 + Ni2(SO4)3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C2H4O4 + O2 = CO2 + H2O"}], "ideal": ["C2H4O4 + O2 = 2CO2 + 2H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C6H12O6 + O2 = H2O + CO2"}], "ideal": ["C6H12O6 + 6O2 = 6H2O + 6CO2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "H2O + CO2 = C7H8 + O2"}], "ideal": ["4H2O + 7CO2 = C7H8 + 9O2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "(NH4)3PO4 + Pb(NO3)4 = Pb3(PO4)4 + NH4NO3"}], "ideal": ["4(NH4)3PO4 + 3Pb(NO3)4 = Pb3(PO4)4 + 12NH4NO3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "BF3 + Li2SO3 = B2(SO3)3 + LiF"}], "ideal": ["2BF3 + 3Li2SO3 = B2(SO3)3 + 6LiF"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C7H17 + O2 = CO2 + H2O"}], "ideal": ["4C7H17 + 45O2 = 28CO2 + 34H2O"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Ag2S = Ag + S8"}], "ideal": ["8Ag2S = 16Ag + S8"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "KOH + Co3(PO4)2 = K3PO4 + Co(OH)2"}], "ideal": ["6KOH + Co3(PO4)2 = 2K3PO4 + 3Co(OH)2"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "Al + HCl = H2 + AlCl3"}], "ideal": ["2Al + 6HCl = 3H2 + 2AlCl3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "AlBr3 + K2SO4 = KBr + Al2(SO4)3"}], "ideal": ["2AlBr3 + 3K2SO4 = 6KBr + Al2(SO4)3"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "V2O5 + CaS = CaO + V2S5"}], "ideal": ["V2O5 + 5CaS = 5CaO + V2S5"]}
+{"input": [{"role": "system", "content": "You are ChemistGPT, can help user balance chemical equation. User for example, if user's input is \"C6H5COOH + O2 = CO2 + H2O\", you will reply the balanced chemical equation: \"2C6H5COOH + 15O2 = 14CO2 + 6H2O\", without explanation. If you can't balance the equation, just reply \"Unknown\""}, {"role": "user", "content": "C2H6 + O2 = CO2 + H2O"}], "ideal": ["2C2H6 + 7O2 = 4CO2 + 6H2O"]}

evals/evals/registry/data/belarusian_lexicon/samples.jsonl

@@ -0,0 +1,300 @@
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "абвязкою"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "абвязкаю"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "абласці"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вобласці"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "абмяну"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "абмену"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "абоўязак"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "абавязак"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "аднасінькіх"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "аднюсенькіх"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "аролі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "арэлі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "асвятленыя"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "асветленыя"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "афармілі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "аформілі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "афісе"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "офісе"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "багаццём"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "багаццем"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "безпечна"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "бяспечна"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "бядамі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "бедамі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "бялыя"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "белыя"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ваблівах"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ваблівых"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вежыкі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вожыкі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "веравыя"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "верхавыя"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вёская"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вясковая"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ветрыстым"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вятрыстым"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вечарыня"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вечарына"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вырастыць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вырастуць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вягняного"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вогненнага"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "газоў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "газаў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гандлёўлі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гандлю"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гарадзе"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "горадзе"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гарманіі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гармоніі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гарызанты"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гарызонты"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "годаў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гадоў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "годы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гады"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "горадаў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гарадоў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гэтам"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гэтым"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "даждж"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "дождж"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "дамоўляемся"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "дамаўляемся"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "домах"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "дамах"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "доступных"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "даступных"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "дрыбы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "грыбы"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "езда"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "язда"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "жаўтні"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "жаўткі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "жаўтыя"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "жоўтыя"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "журавіных"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "журавінных"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "з'езджаў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "з'язджаў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "з'яўілася"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "з'явілася"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "займацься"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "займацца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "заняткоў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "заняткаў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "звук"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гук"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "злучанне"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "злучэнне"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "злучэні"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "злучэнні"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "зміняцца"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "зменяцца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "знаходзіц"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "знаходзіць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "знойдуцься"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "знойдуцца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "зосумаваць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "засумаваць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "історый"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "гісторый"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "кабінце"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "кабінеце"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "кавалком"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "кавалкам"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "калеру"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "колеру"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "калюшня"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "канюшня"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "каляров"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "каляровы"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "каралевскай"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "каралеўскай"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "караляўскага"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "каралеўскага"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "карамелю"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "карамеллю"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "каропам"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "кропам"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "квечка"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "квет��а"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "кінасале"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "кіназале"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "колонак"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "калонак"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лесаў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лясоў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "летаць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лётаць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ліставай"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ліставой"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лістопад"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лістапад"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лясу"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лесу"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лято"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "лета"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "магуць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "могуць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "майстарня"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "майстэрня"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "масштабны"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "маштабны"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "медведзі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "мядзведзі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "мімо"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "міма"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "мнага"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "многа"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "множства"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "мноства"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "можаць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "можа"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "морознай"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "марознай"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "мудрый"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "мудры"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "накрыўалася"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "накрывалася"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "науки"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "навукі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "негатывнага"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "негатыўнага"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "недаўна"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "нядаўна"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "непацярплівая"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "нецярплівая"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "нязалежнасці"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "незалежнасці"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "няўзможна"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "няможна"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "озёр"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "азёр"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паветранне"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паветра"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "павечараць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "павячэраць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пагаршыўся"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пагоршыўся"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пагаршыць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пагоршыць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "падарабленая"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "падробленая"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пазбегаць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пазбягаць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пазірк"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "позірк"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "палепшэння"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паляпшэння"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "палітрой"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "палітрай"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "парахад"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "параход"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паркаўка"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паркоўка"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пасвіцілаў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пасвяціла"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паспехавае"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паспяховае"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пастаўў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "паставіў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "постаяннага"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "пастаяннага"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "празыўна"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прызыўна"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "практыкавацься"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "практыкавацца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прапрадзіць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "праводзіць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "працой"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "працай"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "працягом"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "працягам"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "просторы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прасторы"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прывабляюць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прывабліваюць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прыгажось"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прыгажосць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прыгарадай"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прыгарадам"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прычаплівае"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "прычэплівае"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "птушачы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "птушыны"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "районе"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "раёне"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "распаўсярджваецца"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "распаўсюджваецца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "распеваюць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "распяваюць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "растліны"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "расліны"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "расцутаюць"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "расцвітаюць"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "рашающым"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "рашаючым"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "роблю"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "раблю"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "робоце"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "рабоце"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "светлое"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "светлае"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "серадзіне"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сярэдзіне"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "скласты"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "скласці"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сконцана"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "скончана"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сніданку"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сняданку"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сплатаецца"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сплятаецца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "станоўцца"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "становіцца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "страв"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "страў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сувязана"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "звязана"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сустрэл"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сустрэў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сцянамі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "сценамі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "таленам"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "талентам"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "тварчы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "творчы"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "трабуе"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "патрабуе"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "увасобляючы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "увасабляючы"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "унутранніх"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "унутраных"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "урадзейная"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ураджайная"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "успех"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "поспех"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "учацца"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "вучацца"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "харвац"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "харват"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "хворабы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "хваробы"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "хмарой"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "хмарай"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "царкваў"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "цэркваў"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "цяплы"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "цёплы"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "цячыню"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "цячэнне"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "чаўкі"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "чайкі"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "червоную"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "чырвоную"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ядэрнага"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "ядзернага"}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "якіясці"}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You will be prompted with a single word. Does this word exist in Belarusian language? Answer Y or N."}, {"role": "user", "content": "якіясьці"}], "ideal": "Y"}

evals/evals/registry/data/bigrams/samples.jsonl

@@ -0,0 +1,200 @@
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I'm worried by the fact that my daughter looks to the local carpet seller as a role model."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He found rain fascinating yet unpleasant."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The near-death experience brought new ideas to light."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Separation anxiety is what happens when you can't find your phone."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He realized there had been several deaths on this road, but his concern rose when he saw the exact number."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's no reason a hula hoop can't also be a circus ring."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I've traveled all around Africa and still haven't found the gnu who stole my scarf."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Sometimes, all you need to do is completely make an ass of yourself and laugh it off to realise that life isn’t so bad after all."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The knives were out and she was sharpening hers."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's probably enough glass in my cupboard to build an undersea aquarium."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"One small action would change her life, but whether it would be for better or for worse was yet to be determined."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The estate agent quickly marked out his territory on the dance floor."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Chocolate covered crickets were his favorite snack."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The old rusted farm equipment surrounded the house predicting its demise."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Tomatoes make great weapons when water balloons aren’t available."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He had accidentally hacked into his company's server."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Jason didn’t understand why his parents wouldn’t let him sell his little sister at the garage sale."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Her life in the confines of the house became her new normal."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At that moment I was the most fearsome weasel in the entire swamp."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The doll spun around in circles in hopes of coming alive."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The miniature pet elephant became the envy of the neighborhood."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"For the 216th time, he said he would quit drinking soda after this last Coke."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was the type of guy who liked Christmas lights on his house in the middle of July."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She felt that chill that makes the hairs on the back of your neck when he walked into the room."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Baby wipes are made of chocolate stardust."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The dead trees waited to be ignited by the smallest spark and seek their revenge."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Little Red Riding Hood decided to wear orange today."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"His mind was blown that there was nothing in space except space itself."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I know many children ask for a pony, but I wanted a bicycle with rockets strapped to it."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was an introvert that extroverts seemed to love."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I caught my squirrel rustling through my gym bag."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Homesickness became contagious in the young campers' cabin."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It was a really good Monday for being a Saturday."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At last"}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Greetings from the galaxy MACS0647-JD, or what we call home."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She found his complete dullness interesting."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It must be five o'clock somewhere."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Toddlers feeding raccoons surprised even the seasoned park ranger."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Had he known what was going to happen, he would have never stepped into the shower."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I ate a sock because people on the Internet told me to."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"You should never take advice from someone who thinks red paint dries quicker than blue paint."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He took one look at what was under the table and noped the hell out of there."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Henry couldn't decide if he was an auto mechanic or a priest."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The blinking lights of the antenna tower came into focus just as I heard a loud snap."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"That is an appealing treasure map that I can't read."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Patricia found the meaning of life in a bowl of Cheerios."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Be careful with that butter knife."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He turned in the research paper on Friday; otherwise, he would have not passed the class."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"David proudly graduated from high school top of his class at age 97."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ng'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The paintbrush was angry at the color the artist chose to use."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I'm worried by the fact that my daughter looks to the local carpet seller as a role model."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He found rain fascinating yet unpleasant."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The near-death experience brought new ideas to light."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Separation anxiety is what happens when you can't find your phone."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He realized there had been several deaths on this road, but his concern rose when he saw the exact number."}],"ideal":"4"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's no reason a hula hoop can't also be a circus ring."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I've traveled all around Africa and still haven't found the gnu who stole my scarf."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Sometimes, all you need to do is completely make an ass of yourself and laugh it off to realise that life isn’t so bad after all."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The knives were out and she was sharpening hers."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's probably enough glass in my cupboard to build an undersea aquarium."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"One small action would change her life, but whether it would be for better or for worse was yet to be determined."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The estate agent quickly marked out his territory on the dance floor."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Chocolate covered crickets were his favorite snack."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The old rusted farm equipment surrounded the house predicting its demise."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Tomatoes make great weapons when water balloons aren’t available."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He had accidentally hacked into his company's server."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Jason didn’t understand why his parents wouldn’t let him sell his little sister at the garage sale."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Her life in the confines of the house became her new normal."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At that moment I was the most fearsome weasel in the entire swamp."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The doll spun around in circles in hopes of coming alive."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The miniature pet elephant became the envy of the neighborhood."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"For the 216th time, he said he would quit drinking soda after this last Coke."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was the type of guy who liked Christmas lights on his house in the middle of July."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She felt that chill that makes the hairs on the back of your neck when he walked into the room."}],"ideal":"5"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Baby wipes are made of chocolate stardust."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The dead trees waited to be ignited by the smallest spark and seek their revenge."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Little Red Riding Hood decided to wear orange today."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"His mind was blown that there was nothing in space except space itself."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I know many children ask for a pony, but I wanted a bicycle with rockets strapped to it."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was an introvert that extroverts seemed to love."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I caught my squirrel rustling through my gym bag."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Homesickness became contagious in the young campers' cabin."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It was a really good Monday for being a Saturday."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At last"}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Greetings from the galaxy MACS0647-JD, or what we call home."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She found his complete dullness interesting."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It must be five o'clock somewhere."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Toddlers feeding raccoons surprised even the seasoned park ranger."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Had he known what was going to happen, he would have never stepped into the shower."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I ate a sock because people on the Internet told me to."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"You should never take advice from someone who thinks red paint dries quicker than blue paint."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He took one look at what was under the table and noped the hell out of there."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Henry couldn't decide if he was an auto mechanic or a priest."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The blinking lights of the antenna tower came into focus just as I heard a loud snap."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"That is an appealing treasure map that I can't read."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Patricia found the meaning of life in a bowl of Cheerios."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Be careful with that butter knife."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He turned in the research paper on Friday; otherwise, he would have not passed the class."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"David proudly graduated from high school top of his class at age 97."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'th'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The paintbrush was angry at the color the artist chose to use."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I'm worried by the fact that my daughter looks to the local carpet seller as a role model."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He found rain fascinating yet unpleasant."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The near-death experience brought new ideas to light."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Separation anxiety is what happens when you can't find your phone."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He realized there had been several deaths on this road, but his concern rose when he saw the exact number."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's no reason a hula hoop can't also be a circus ring."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I've traveled all around Africa and still haven't found the gnu who stole my scarf."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Sometimes, all you need to do is completely make an ass of yourself and laugh it off to realise that life isn’t so bad after all."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The knives were out and she was sharpening hers."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's probably enough glass in my cupboard to build an undersea aquarium."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"One small action would change her life, but whether it would be for better or for worse was yet to be determined."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The estate agent quickly marked out his territory on the dance floor."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Chocolate covered crickets were his favorite snack."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The old rusted farm equipment surrounded the house predicting its demise."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Tomatoes make great weapons when water balloons aren’t available."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He had accidentally hacked into his company's server."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Jason didn’t understand why his parents wouldn’t let him sell his little sister at the garage sale."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Her life in the confines of the house became her new normal."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At that moment I was the most fearsome weasel in the entire swamp."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The doll spun around in circles in hopes of coming alive."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The miniature pet elephant became the envy of the neighborhood."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"For the 216th time, he said he would quit drinking soda after this last Coke."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was the type of guy who liked Christmas lights on his house in the middle of July."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She felt that chill that makes the hairs on the back of your neck when he walked into the room."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Baby wipes are made of chocolate stardust."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The dead trees waited to be ignited by the smallest spark and seek their revenge."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Little Red Riding Hood decided to wear orange today."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"His mind was blown that there was nothing in space except space itself."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I know many children ask for a pony, but I wanted a bicycle with rockets strapped to it."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was an introvert that extroverts seemed to love."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I caught my squirrel rustling through my gym bag."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Homesickness became contagious in the young campers' cabin."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It was a really good Monday for being a Saturday."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At last"}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Greetings from the galaxy MACS0647-JD, or what we call home."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She found his complete dullness interesting."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It must be five o'clock somewhere."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Toddlers feeding raccoons surprised even the seasoned park ranger."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Had he known what was going to happen, he would have never stepped into the shower."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I ate a sock because people on the Internet told me to."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"You should never take advice from someone who thinks red paint dries quicker than blue paint."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He took one look at what was under the table and noped the hell out of there."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Henry couldn't decide if he was an auto mechanic or a priest."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The blinking lights of the antenna tower came into focus just as I heard a loud snap."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"That is an appealing treasure map that I can't read."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Patricia found the meaning of life in a bowl of Cheerios."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Be careful with that butter knife."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He turned in the research paper on Friday; otherwise, he would have not passed the class."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"David proudly graduated from high school top of his class at age 97."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ai'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The paintbrush was angry at the color the artist chose to use."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I'm worried by the fact that my daughter looks to the local carpet seller as a role model."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He found rain fascinating yet unpleasant."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The near-death experience brought new ideas to light."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Separation anxiety is what happens when you can't find your phone."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He realized there had been several deaths on this road, but his concern rose when he saw the exact number."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's no reason a hula hoop can't also be a circus ring."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I've traveled all around Africa and still haven't found the gnu who stole my scarf."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Sometimes, all you need to do is completely make an ass of yourself and laugh it off to realise that life isn’t so bad after all."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The knives were out and she was sharpening hers."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"There's probably enough glass in my cupboard to build an undersea aquarium."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"One small action would change her life, but whether it would be for better or for worse was yet to be determined."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The estate agent quickly marked out his territory on the dance floor."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Chocolate covered crickets were his favorite snack."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The old rusted farm equipment surrounded the house predicting its demise."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Tomatoes make great weapons when water balloons aren’t available."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He had accidentally hacked into his company's server."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Jason didn’t understand why his parents wouldn’t let him sell his little sister at the garage sale."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Her life in the confines of the house became her new normal."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At that moment I was the most fearsome weasel in the entire swamp."}],"ideal":"2"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The doll spun around in circles in hopes of coming alive."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The miniature pet elephant became the envy of the neighborhood."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"For the 216th time, he said he would quit drinking soda after this last Coke."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was the type of guy who liked Christmas lights on his house in the middle of July."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She felt that chill that makes the hairs on the back of your neck when he walked into the room."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Baby wipes are made of chocolate stardust."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The dead trees waited to be ignited by the smallest spark and seek their revenge."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Little Red Riding Hood decided to wear orange today."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"His mind was blown that there was nothing in space except space itself."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I know many children ask for a pony, but I wanted a bicycle with rockets strapped to it."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He was an introvert that extroverts seemed to love."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I caught my squirrel rustling through my gym bag."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Homesickness became contagious in the young campers' cabin."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It was a really good Monday for being a Saturday."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"At last"}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Greetings from the galaxy MACS0647-JD, or what we call home."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"She found his complete dullness interesting."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"It must be five o'clock somewhere."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Toddlers feeding raccoons surprised even the seasoned park ranger."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Had he known what was going to happen, he would have never stepped into the shower."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"I ate a sock because people on the Internet told me to."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"You should never take advice from someone who thinks red paint dries quicker than blue paint."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He took one look at what was under the table and noped the hell out of there."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Henry couldn't decide if he was an auto mechanic or a priest."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The blinking lights of the antenna tower came into focus just as I heard a loud snap."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"That is an appealing treasure map that I can't read."}],"ideal":"3"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Patricia found the meaning of life in a bowl of Cheerios."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"Be careful with that butter knife."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"He turned in the research paper on Friday; otherwise, he would have not passed the class."}],"ideal":"1"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"David proudly graduated from high school top of his class at age 97."}],"ideal":"0"}
+{"input":[{"role":"system","content":"You will be presented with a sentence. The task is to count the frequency of the bigram 'ea'. After reading the sentence tell me the number of times the bigram appeared by saying 'X' where 'X' is the frequency."},{"role":"user","content":"The paintbrush was angry at the color the artist chose to use."}],"ideal":"0"}

evals/evals/registry/data/born_first/born_first.jsonl

@@ -0,0 +1,122 @@
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Abraham Lincoln born before Franklin Pierce? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Abraham Lincoln born before Andrew Johnson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Andrew Jackson born before John Quincy Adams? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Andrew Jackson born before William Harrison? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Andrew Johnson born before Franklin Pierce? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Andrew Johnson born before Abraham Lincoln? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Benjamin Harrison born before James Garfield? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Benjamin Harrison born before Chester A. Arthur? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Benjamin Harrison born before Grover Cleveland? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Benjamin Harrison born before Grover Cleveland? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Bill Clinton born before George W. Bush? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Bill Clinton born before Donald Trump? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Bill Clinton born before Joe Biden? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Calvin Coolidge born before Warren Harding? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Calvin Coolidge born before Herbert Hoover? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Chester A. Arthur born before Ulysses S. Grant? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Chester A. Arthur born before Rutherford B. Hayes? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Chester A. Arthur born before James Garfield? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Chester A. Arthur born before Grover Cleveland? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Chester A. Arthur born before Benjamin Harrison? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Chester A. Arthur born before Grover Cleveland? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Donald Trump born before Bill Clinton? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Donald Trump born before George W. Bush? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Donald Trump born before Joe Biden? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Dwight Eisenhower born before Harry Truman? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Franklin Pierce born before Millard Fillmore? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Franklin Pierce born before Abraham Lincoln? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Franklin Pierce born before Andrew Johnson? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Franklin Roosevelt born before Herbert Hoover? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Franklin Roosevelt born before Harry Truman? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was George H.W. Bush born before John F. Kennedy? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was George H.W. Bush born before Jimmy Carter? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was George W. Bush born before Bill Clinton? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was George W. Bush born before Donald Trump? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was George W. Bush born before Joe Biden? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was George Washington born before John Adams? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Gerald Ford born before John F. Kennedy? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Gerald Ford born before Lyndon Johnson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Gerald Ford born before Richard Nixon? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Gerald Ford born before Ronald Reagan? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before James Garfield? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before James Garfield? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before Chester A. Arthur? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before Chester A. Arthur? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before Benjamin Harrison? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before Grover Cleveland? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before Grover Cleveland? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before William McKinley? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before Benjamin Harrison? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Grover Cleveland born before William McKinley? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Harry Truman born before Franklin Roosevelt? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Harry Truman born before Dwight Eisenhower? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Herbert Hoover born before Calvin Coolidge? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Herbert Hoover born before Franklin Roosevelt? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Buchanan born before John Tyler? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Buchanan born before James Polk? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Buchanan born before Zachary Taylor? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Garfield born before Chester A. Arthur? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Garfield born before Grover Cleveland? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Garfield born before Benjamin Harrison? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Garfield born before Grover Cleveland? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Madison born before Thomas Jefferson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Madison born before James Monroe? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Monroe born before James Madison? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Polk born before John Tyler? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Polk born before Millard Fillmore? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was James Polk born before James Buchanan? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Jimmy Carter born before John F. Kennedy? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Jimmy Carter born before George H.W. Bush? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Joe Biden born before Bill Clinton? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Joe Biden born before George W. Bush? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Joe Biden born before Donald Trump? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Adams born before George Washington? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Adams born before Thomas Jefferson? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John F. Kennedy born before Richard Nixon? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John F. Kennedy born before Gerald Ford? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John F. Kennedy born before Jimmy Carter? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John F. Kennedy born before Ronald Reagan? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John F. Kennedy born before George H.W. Bush? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Quincy Adams born before Andrew Jackson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Quincy Adams born before William Harrison? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Tyler born before Martin Van Buren? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Tyler born before James Polk? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Tyler born before Zachary Taylor? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was John Tyler born before James Buchanan? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Lyndon Johnson born before Richard Nixon? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Lyndon Johnson born before Gerald Ford? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Lyndon Johnson born before Ronald Reagan? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Martin Van Buren born before John Tyler? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Martin Van Buren born before Zachary Taylor? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Millard Fillmore born before James Polk? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Millard Fillmore born before Franklin Pierce? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Richard Nixon born before John F. Kennedy? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Richard Nixon born before Lyndon Johnson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Richard Nixon born before Gerald Ford? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Richard Nixon born before Ronald Reagan? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Ronald Reagan born before John F. Kennedy? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Ronald Reagan born before Lyndon Johnson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Ronald Reagan born before Richard Nixon? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Ronald Reagan born before Gerald Ford? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Rutherford B. Hayes born before Ulysses S. Grant? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Rutherford B. Hayes born before Chester A. Arthur? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Theodore Roosevelt born before William Taft? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Theodore Roosevelt born before Woodrow Wilson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Theodore Roosevelt born before Warren Harding? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Thomas Jefferson born before John Adams? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Thomas Jefferson born before James Madison? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Ulysses S. Grant born before Rutherford B. Hayes? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Ulysses S. Grant born before Chester A. Arthur? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Warren Harding born before Theodore Roosevelt? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Warren Harding born before Calvin Coolidge? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was William Harrison born before John Quincy Adams? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was William Harrison born before Andrew Jackson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was William McKinley born before Grover Cleveland? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was William McKinley born before Grover Cleveland? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was William Taft born before Theodore Roosevelt? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was William Taft born before Woodrow Wilson? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Woodrow Wilson born before Theodore Roosevelt? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Woodrow Wilson born before William Taft? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Zachary Taylor born before Martin Van Buren? Answer Y or N."}], "ideal": "N"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Zachary Taylor born before John Tyler? Answer Y or N."}], "ideal": "Y"}
+{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Was Zachary Taylor born before James Buchanan? Answer Y or N."}], "ideal": "Y"}

evals/evals/registry/data/chess/match.jsonl

@@ -0,0 +1,101 @@
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2r3k1/p1q2ppp/1p3n2/1Q2p3/2P5/6P1/P4PKP/3R4 Possible Moves: A: f4, B: Rd2, C: Kg1, D: Qb2 Answer only with the letter of the beset move."}], "ideal": "D"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bq1rk1/pp5p/3pp1p1/2p1b2p/2P1P2P/3PQ3/PP3P2/RNB2RK1 Possible Moves: A: h8, B: Qxh4, C: Kg7, D: Bxb2 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r12k5/4b3/1p4p1/p3P2p/PP3P1P/2P2KP1/8/8 Possible Moves: A: Ke4, B: g4, C: c3, D: bxa Answer only with the letter of the beset move."}], "ideal": "D"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 4r1k1/5pp1/2Nq3p/3PnQ2/8/6PP/5P2/3R2K1 Possible Moves: A: Qf4, B: Kh1, C: Nxe5, D: Rd3 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r3r1k1/p2QBp2/1p4pp/2p1b3/2q1P3/P1P5/1P3PPP/3RR1K1 Possible Moves: A: Qxe8, B: Kh1, C: Qb3, D: Rd3 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 4rqk1/ppp2pp1/5nnp/2b1r3/4N3/P1P2Q1P/1PB2PP1/R1B1R1K1 Possible Moves: A: Nh4, B: R5e6, C: Bxh6, D: Rd1 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r3k2r/1b2ppb1/pq1p3p/1pn4P/4P1n1/P1NBBN2/1PP1Q1P1/3R1RK1 Possible Moves: A: Nd5, B: Bxc5, C: g3, D: Bd4 Answer only with the letter of the beset move."}], "ideal": "D"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 2r2bkr/p5p1/1q2pBpp/2p1P3/2Pp1R2/5Q2/PP3PPP/R5K1 Possible Moves: A: Kh7, B: gxf6, C: g3, D: Qe3 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 2r2bkr/p7/1q2pRpp/2p1P3/2Pp4/5Q2/PP3PPP/R5K1 Possible Moves: A: Qxb2, B: Bg7, C: Rh7, D: a6 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 2r4r/p4Qbk/1q2p1Rp/2p1P3/2Pp4/8/PP3PPP/R5K1 Possible Moves: A: Rhg8, B: a6, C: h5, D: d3 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r5k1/4ppb1/p2p3p/1p5P/1P6/P1R2N2/2P2KP1/8 Possible Moves: A: Rd8, B: Bh8, C: h5, D: Bxc3 Answer only with the letter of the beset move."}], "ideal": "D"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r3r1k1/4ppb1/pn1p3p/1p1P3P/1P6/P2B1N2/2P3P1/3RR1K1 Possible Moves: A: Kf1, B: Be4, C: a4, D: Ra1 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r1bqkbnr/pp3ppp/2n1p3/3pP3/3p4/2P2N2/PP3PPP/RNBQKB1R Possible Moves: A: cxd4, B: h3, C: a3, D: Qd2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 2r1k2r/1p2bppp/pq2p3/1B1pPn2/1n1P2P1/P4N2/5P1P/RQB1K2R Possible Moves: A: Qxb5, B: Nc6, C: Kf8, D: Qd2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2QR4/p4ppk/1p3n2/4p2p/q7/6P1/5PKP/8 Possible Moves: A: Qf5, B: Qc3, C: h4, D: f3 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: rnbqkb1r/pp2pppp/5n2/3P4/Q7/8/PP1P1PPP/RNB1KBNR Possible Moves: A: Bc3, B: Nfd7, C: Nbd7, D: b5 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 3rr1k1/pq3ppp/1p4nb/2pp1B1N/b2P2QP/P3P1B1/1P3PP1/3R1RK1 Possible Moves: A: Rb1, B: b3, C: Nxg7, D: Qxg6 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 3rr1k1/p1R1nppp/qp5b/1b1p3N/3Q3P/P3P1B1/1P3PP1/1BR3K1 Possible Moves: A: Qc8, B: Qa4, C: Bd7, D: Ng6 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "white to move FEN: rnbq1rk1/ppp1ppbp/6p1/8/3PP3/2P2N2/P4PPP/R1BQKB1R Possible Moves: A: Be2, B: a3, C: h3, D: Ke2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bq1rk1/pp1nbppp/3p2n1/2pP4/8/1PN3P1/PB2NPBP/R2Q1RK1 Possible Moves: A: Re8, B: a6, C: h6, D: f6 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 5r2/ppp1p2k/4B1p1/4B2p/8/2P5/P5PP/5RK1 Possible Moves: A: Rxf1, B: Rd8, C: Rg8, D: Rh8 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 6rr/pk6/1pp3b1/2n1p1Bn/1PP3pP/2N5/P4PB1/2KR3R Possible Moves: A: Nd3+, B: Rf8, C: Rf7, D: Rf8 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bqkb1r/pppp1ppp/5nn1/3P4/2P1Pp2/2N2N2/PP4PP/R1BQKB1R Possible Moves: A: Bc5, B: Bb4, C: fxe3, D: h6 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bqkb1r/ppppnppp/5n2/3P4/2P2p2/2N2N2/PP2P1PP/R1BQKB1R b KQkq - 1 6 Possible Moves: A: d6, B: b5, C: a6, D: Ng6 Answer only with the letter of the beset move."}], "ideal": "D"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r1bqkb1r/pppp1ppp/5nn1/3P4/2P2p2/2N2N2/PP2P1PP/R1BQKB1R Possible Moves: A: Qb3, B: e3, C: a6, D: Bb4 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r1b2rk1/pppp1ppp/6n1/3P4/2P2q2/2P2B1P/P2Q2P1/R4K1R Possible Moves: A: Ke1, B: Qxf4, C: Kg1, D: g3 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r1bqkbnr/pp1p1ppp/6n1/2pPp3/4PP2/2N5/PPP3PP/R1BQKBNR Possible Moves: A: d6, B: f5, C: h4, D: fxe5 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR Possible Moves: A: e4, B: h4, C: g4, D: Nh3 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r7/1b1k2q1/p1p5/1pb5/4Q3/8/PPP2P2/1K1R4 Possible Moves: A: Qd4, B: Bd4, C: Kc8, D: Kc7 Answer only with the letter of the beset move."}], "ideal": "D"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 2rqr3/1b3pbk/pp2p1np/3pP1P1/3P4/4QNP1/PP3PN1/R1R2BK1 Possible Moves: A: hxg5, B: Rc4, C: Rh8, D: h5 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r3kr2/ppp1q1b1/3npn2/7p/2P1N1pB/1PN3P1/P1Q3BP/3R1RK1 Possible Moves: A: Bh6, B: a6, C: Rd8, D: O-O-O Answer only with the letter of the beset move."}], "ideal": "D"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 2kr1r2/ppp1q1b1/4pn2/2P4p/4N1pB/1P4P1/P1Q3BP/3R1RK1 Possible Moves: A: Rxd1, B: Rd7, C: Rd5, D: Rd2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2kr4/ppq5/2p1pB2/2P4p/6p1/1P4P1/P1Q3BP/5RK1 Possible Moves: A: Bxd8, B: Qe4, C: Qc4 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r7/pb2R3/1p3pkp/2p3p1/2Pp4/PP1N1P2/2P3PP/6K1 Possible Moves: A: Kf2, B: h3, C: Rxb7 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 8/pR6/1p3pkp/2p3p1/2Pp4/PP1N1P2/2P1rKPP/8 Possible Moves: A: Kg1, B: Kg3, C: Kxe2 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r7/pR6/1p3pkp/2p3p1/2Pp4/PP1N1P2/2P3PP/6K1 Possible Moves: A: a6, B: h5, C: a5 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r3r3/pb3pkp/1p3np1/2p3B1/2Pp4/PP1N4/2P2PPP/R3R1K1 Possible Moves: A: Ne4, B: a6, C: a5 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqkb1r/pp2pp1p/2p2np1/3p4/2P5/1P3NP1/P2PPP1P/RNBQKB1R Possible Moves: A: Bb2, B: e3, C: h3 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqk2r/p1p1npbp/1p1pp1p1/8/3P4/4PNP1/PPP2PBP/RNBQ1RK1 Possible Moves: A: c4, B: Re1, C: e4 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2rq1r2/1b1nnpbk/pp2p1pp/3pP3/3P4/4BNPP/PPNQ1PB1/R3R1K1 Possible Moves: A: Bf4, B: Rad1, C: Qd1 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2rq1r2/1b1n1pbk/pp2p1pp/3pP3/3P4/4nNPP/PP1Q1PB1/R1R1N1K1 Possible Moves: A: Rxc8, B: Nd3, C: h4 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2r1r1k1/1b3p2/pp2p1qb/3pP3/3P1P2/4Q1P1/PP6/R1R2BK1 Possible Moves: A: Re1, B: b3, C: Bd3 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2r4r/1b2kpq1/p3p3/1p1pPP1R/3P4/3Bb1P1/PP3K2/7R Possible Moves: A: Kf3, B: Kf1, C: Ke2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 7r/1b3p2/p3pk2/1p1p3R/3P4/3BK1P1/PP6/8 Possible Moves: A: Re5, B: Rh7, C: Rxh8 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 8/3bkp2/pR2p3/3p4/Pp1P4/3BK1P1/1P6/8 Possible Moves: A: b3, B: Rxb4, C: Rb7 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r3r1k1/pp3ppp/8/2Q5/4p1Pq/1P1RP2P/P4PB1/4n1K1 Possible Moves: A: Rd5, B: Rd4, C: Rd7 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqkb1r/ppp2ppp/4pn2/3p4/3PP3/5N2/PPP2PPP/RNBQKB1R Possible Moves: A: Be3, B: e5, C: Nc3 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqkb1r/pp1n1ppp/4p3/3pP3/3p4/2PB1N2/PP3PPP/RNBQK2R Possible Moves: A: Bf4, B: cxd4, C: Bg5 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bqk2r/pp1n1ppp/2n1p3/3pP3/3P4/1PbB1N2/P2B1PPP/R2QK2R Possible Moves: A: Ba5, B: Bxa1, C: Bb4 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bq1rk1/1p3ppp/p3p3/3pP3/P1n5/3B3Q/5PPP/2R2RK1 Possible Moves: A: Re8, B: h6, C: f5 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r2q1rk1/1p1b1p1p/p3p1p1/4P3/P6R/7Q/5PPP/5RK1 Possible Moves: A: Qg5, B: Qc7, C: Qb8 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r3r1k1/3Q2pp/1p2p3/4Pp2/1pP1b3/1P6/P4PPP/3R1BK1 Possible Moves: A: Kh8, B: f4, C: Rxa2 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r3r1k1/6pp/1p2p3/1Q2Pb2/1pP2p2/1P3P2/P5PP/3R1BK1 Possible Moves: A: Reb8, B: Rxa2, C: Re7 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 6k1/6pp/4p3/2P1Pb2/1Q3p2/1P3P1P/r5P1/5BK1 Possible Moves: A: Ra1, B: Rxa2, C: h6 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 8/6pp/2PQp1k1/4Pb2/5p2/1P3P1P/r5P1/5BK1 Possible Moves: A: Kh6, B: Ra8, C: Kh5 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: rnbqkb1r/pp3ppp/4pn2/3p2B1/2PN4/2N5/PP2PPPP/R2QKB1R Possible Moves: A: e5, B: Bc5, C: h6 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bqk2r/pp2bpp1/2N1pn1p/3p4/2P4B/2N1P3/PP3PPP/R2QKB1R Possible Moves: A: e5, B: Qd7, C: bxc6 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r3r1k1/3Q2pp/1p2p3/4P3/1pP1bp2/1P3P2/P5PP/3R1BK1 Possible Moves: A: Kc8, B: h6, C: Bf5 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 6k1/6pp/4p3/4Pb2/1QP2p2/1P3P1P/P2r2P1/5BK1 Possible Moves: A: Rd1, B: h6, C: Rxa2 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 6k1/6pp/4p3/4Pb2/1QP2p2/1P3P1P/P2r2P1/5BK1 Possible Moves: A: Rd3, B: Rd7, C: Rd1 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqkb1r/ppp2ppp/4pn2/3p4/3P4/3BPN2/PPP2PPP/RNBQK2R Possible Moves: A: a6, B: b6, C: c5 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: rnbq1rk1/p1p1bppp/1p2pn2/3p4/3P4/1P1BPN2/PBP2PPP/RN1Q1RK1 Possible Moves: A: a6, B: Nbd7, C: c5 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r2q1rk1/pbpnbppp/1p2pn2/3pN3/3P4/1P1BP3/PBPN1PPP/R2Q1RK1 Possible Moves: A: Rc8, B: a5, C: a6 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r2q1rk1/pb2bppp/1p2pn2/3pP3/3P4/1P1B4/PBPN2PP/R2Q1RK1 Possible Moves: A: a6, B: Ne4, C: a5 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r2q1rk1/pb2bppp/1p2p3/4P3/3Pp3/1P6/PBP1B1PP/R2Q1RK1 Possible Moves: A: a6, B: Bg5, C: a5 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 5rk1/p5pp/1pb1p3/4P3/2P3PP/PP6/1B6/6K1 Possible Moves: A: Rf7, B: Rf4, C: Bb7 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqkbnr/2p1pppp/p7/1p6/8/2N1PN2/PP1P1PPP/R1BQKB1R Possible Moves: A: d4, B: b4, C: Ne4 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rnbqk2r/ppp2pp1/3bpn1p/8/2pP4/2N2N2/PP1BPPPP/R2QKB1R Possible Moves: A: d4, B: e4, C: Rc1 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r4rk1/1p4p1/p1nqp2p/3p1pn1/3P4/1BB1Pb1P/PPR2PP1/2RQ3K Possible Moves: A: gxf3, B: Qf1, C: Qg1 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 6nk/1p4q1/p3p2p/3p1p1Q/3P1Pr1/1BB1P3/PP5R/7K Possible Moves: A: Qg7, B: Qg6, C: Rg2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r1b2rk1/1pp1qpp1/p1nbpn1p/1N6/2BP4/4PN1P/PP1B1PP1/R2QK2R Possible Moves: A: Qg7, B: Nc3, C: Nxd6 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 6nk/1p2q3/p3p2p/3p1p1Q/3P1Pr1/1BB1P3/PPR5/7K Possible Moves: A: Ba5, B: Nc3, C: Be1 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r4r1k/2R5/7p/pp4b1/3P4/1PP4P/P7/4R2K Possible Moves: A: Ba5, B: Re2, C: Re4 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 8/2k5/2P4p/p2Pb3/r7/4K2P/2R5/8 Possible Moves: A: h4, B: Re2, C: Rg2 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 8/2k5/2P4K/1r5P/8/8/pbR5/8 Possible Moves: A: h4, B: Rb6, C: a1=Q Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 4r1k1/2qb2p1/p4n1p/1pb5/2pQ3B/2N4P/PPP2PP1/4R1K1 Possible Moves: A: h4, B: Qe4, C: Rxe8 Answer only with the letter of the beset move."}], "ideal": "C"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 4r3/1rq1pk1p/pp1pRnp1/2pP4/2P2Pp1/2P2B1P/P3Q3/4R1K1 Possible Moves: A: Be4, B: hxg4, C: Rxe8 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 6k1/4R1P1/3p4/3P1K2/2P5/1p6/8/8 Possible Moves: A: Ke6, B: axb3, C: Rxe8 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r2q1rk1/ppp3bp/3nP1p1/8/5B2/6P1/P5BP/1R1Q1R1K Possible Moves: A: Kh8, B: Rb8, C: Rxe8 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: rnbqk2r/ppp1ppbp/3p1np1/8/2PP4/2N3P1/PP2PPBP/R1BQK1NR Possible Moves: A: Nbd7, B: O-O, C: a6 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bq1rk1/ppp1n1bp/3p1np1/3Ppp2/2P1PP2/2N1B1P1/PP1N2BP/R2Q1RK1 Possible Moves: A: Nbd7, B: Ng4, C: fxe4 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bq1rk1/ppp1n1bp/3p2p1/3Pp3/2P1NP2/4B1P1/PP4BP/R2Q1RK1 Possible Moves: A: Nbd7, B: Nf5, C: fxe4 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 3q1rk1/1b4p1/p3B2p/1pp5/3p1PP1/P2P4/1PP4P/4RQK1 Possible Moves: A: Nbd7, B: Kh8, C: Rf7 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 5r1k/1b4p1/p3B2p/1pp2P2/3p2Pq/P2P3P/1PP5/4RQK1 Possible Moves: A: a5, B: Qg3, C: Rf7 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: 3Brrk1/1b4p1/p3pq1p/1pp3N1/3p1PP1/P2P4/BPP4P/4RQK1 Possible Moves: A: a5, B: Qxd8, C: Rf7 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "Black to move FEN: r1bqkb1r/pp3ppp/2n1p3/2pnP3/2Bp1P2/P2P1N2/1PP3PP/RNBQ1RK1 Possible Moves: A: h6, B: a6, C: a5 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r1bqk2r/1p2bppp/p1n1p3/2p1P3/2Bp1P2/P2P1N2/1PPN2PP/R1B2QK1 Possible Moves: A: Qe1, B: a6, C: Qe2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 4rrk1/1b4pp/p1n1pq2/1pp5/3p1PP1/P2P1N2/BPPB3P/4RQK1 Possible Moves: A: Ng5, B: f5, C: Qe2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 5r1k/1b4p1/p3B2p/1pp2P2/3p2Pq/P2P4/1PP4P/4RQK1 Possible Moves: A: Qf2, B: Rd1, C: Qe2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rn1qk3/6b1/1ppPb3/p1P3PR/3pNQ2/6P1/PP4P1/2KR4 Possible Moves: A: Rh7, B: Rd1, C: Rh6 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 8/8/n1p5/p1pk4/3p2Q1/6P1/PP4P1/2KR4 Possible Moves: A: Qd7, B: Rd1, C: Qh3 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rn1qk2r/3b2b1/1ppPp3/p1P3Pp/3PN3/6P1/PP1Q2P1/2KR1B1R Possible Moves: A: Qf4, B: Rd1, C: Bc4 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: r7/3k1P2/n1pPb3/p1p5/3p1Q2/6P1/PP4P1/2KR4 Possible Moves: A: f8=Q, B: Rd1, C: Re1 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 1r1q1rk1/1p2bpp1/1Nnpbn2/4p2p/2P1P3/3P1PP1/4N1BP/1RBQ1RK1 Possible Moves: A: f8=Q, B: f4, C: Rb2 Answer only with the letter of the beset move."}], "ideal": "B"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 1r1q1rk1/1p2bpp1/1Nnpbn2/8/2P1PN1p/3P2P1/6BP/1RBQ1RK1 Possible Moves: A: Nxe6, B: f4, C: Rb2 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 1r1q2k1/1p2b1pn/1N1pp3/4n1P1/2P1P2p/3P3B/7P/1RBQ1rK1 Possible Moves: A: Kxf1, B: Kg2, C: Bxf1 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2k5/3R2K1/3p1N2/5P2/2P5/6bp/8/8 Possible Moves: A: Kf8, B: Kg2, C: Re7 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2k5/3R2K1/3p4/5P2/2P5/8/7b/8 Possible Moves: A: Re7, B: Rxc7, C: Re7 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: 2rr4/2qnbkp1/pp1p1n1p/8/2P5/1PNB1b1P/P3QPPB/2R2RK1 Possible Moves: A: gxf3, B: Qc2, C: Qe1 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rn1qk2r/pp2ppbp/2p2np1/4P3/2B2Qb1/2N5/PPP2PPP/R1B1K1NR Possible Moves: A: Nbd7, B: Qa5, C: h5 Answer only with the letter of the beset move."}], "ideal": "A"}
+{"input": [{"role": "system", "content": "TASK: Read the chess position provided in FEN-notation, then identify the best move to the board position below, in the format A. Your answer should only contain the letter of the correct move. Do not provide any further explanation."}, {"role": "user", "content": "White to move FEN: rn2k2r/pp2ppPp/2p5/6Q1/2q3b1/2N5/PPP2PPP/R1B1K1NR Possible Moves: A: Rg8, B: Qd4, C: Qxc3 Answer only with the letter of the beset move."}], "ideal": "A"}

evals/evals/registry/data/complex_replace_characters/samples.jsonl

@@ -0,0 +1,100 @@
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Hello! How are you?"}], "ideal": ["Hyllo! How zry you?"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["Lyt's sit down znd ynjoy somy winy. Aftyr zll, lify is short!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["Lorym ipsum dolor sit zmyt, consyctytur zdipiscing ylit, syd do yiusmod tympor incididunt ut lzbory yt dolory mzgnz zliquz."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["Wikipydiz is hostyd by thy Wikimydiz Foundztion, z non-profit orgznizztion thzt zlso hosts z rzngy of othyr projycts."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Joe and I updated the QA sheet last night. Huge huge props to everyone - it's literally night and day compared to last week!"}], "ideal": ["Joy znd I updztyd thy QA shyyt lzst night. Hugy hugy props to yvyryony - it's lityrzlly night znd dzy compzryd to lzst wyyk!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If you livy to by z hundryd, I hopy I livy to by z hundryd minus ony dzy, so thzt I nyvyr hzvy to livy z dzy without you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["Doing nothing is vyry hzrd to do. You nyvyr know whyn you'ry finishyd."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["Dyfyzting trzffic is thy ultimzty boss bzttly. Evyn thy most powyrful humzns in thy world cznnot dyfyzt trzffic."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GitHub Codyspzcys offyrs z complyty dyv ynvironmynt in syconds, so you czn cody, build, tyst, znd opyn pull ryquysts from zny rypo znywhyry."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"a\" character is encountered, replace it with \"z\". 2) if \"e\" character is encountered, replace it with \"y\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How zry you?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["Hzvy you hyzrd of thy moviy, Constipztyd? It's not out yyt."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Hello! How are you?"}], "ideal": ["Hellm! Hmw are ymu?"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["Let's sxt dmwn and enjmy smme wxne. After all, lxfe xs shmrt!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["Lmrem xpsum dmlmr sxt amet, cmnsectetur adxpxscxng elxt, sed dm exusmmd tempmr xncxdxdunt ut labmre et dmlmre magna alxqua."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["Wxkxpedxa xs hmsted by the Wxkxmedxa Fmundatxmn, a nmn-prmfxt mrganxzatxmn that alsm hmsts a range mf mther prmjects."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["Sally and I updated the QA sheet thxs mmrnxng. Huge huge prmps tm everymne - xt's lxterally nxght and day cmmpared tm yesterday!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If ymu lxve tm be a hundred, I hmpe I lxve tm be a hundred mxnus mne day, sm that I never have tm lxve a day wxthmut ymu."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["Dmxng nmthxng xs very hard tm dm. Ymu never knmw when ymu're fxnxshed."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["Defeatxng traffxc xs the ultxmate bmss battle. Even the mmst pmwerful humans xn the wmrld cannmt defeat traffxc."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GxtHub Cmdespaces mffers a cmmplete dev envxrmnment xn secmnds, sm ymu can cmde, buxld, test, and mpen pull requests frmm any repm anywhere."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"i\" character is encountered, replace it with \"x\". 2) if \"o\" character is encountered, replace it with \"m\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "Hmw are ymu?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["Have ymu heard mf the mmvxe, Cmnstxpated? It's nmt mut yet."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Hello! How are you?"}], "ideal": ["Hello! How aae you?"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["Let's sit uown anu enjoy some wine. Aftea all, life is shoat!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["Loaem ipsum uoloa sit amet, consectetua auipiscing elit, seu uo eiusmou tempoa inciuiuunt ut laboae et uoloae magna aliqua."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["Wikipeuia is hosteu by the Wikimeuia Founuation, a non-paofit oaganization that also hosts a aange of othea paojects."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["Sally anu I upuateu the QA sheet this moaning. Huge huge paops to eveayone - it's liteaally night anu uay compaaeu to yesteauay!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If you live to be a hunuaeu, I hope I live to be a hunuaeu minus one uay, so that I nevea have to live a uay without you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["Doing nothing is veay haau to uo. You nevea know when you'ae finisheu."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["Defeating taaffic is the ultimate boss battle. Even the most poweaful humans in the woalu cannot uefeat taaffic."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GitHub Couespaces offeas a complete uev enviaonment in seconus, so you can coue, builu, test, anu open pull aequests faom any aepo anywheae."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"d\" character is encountered, replace it with \"u\". 2) if \"r\" character is encountered, replace it with \"a\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How aae you?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["Have you heaau of the movie, Constipateu? It's not out yet."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Hello! How are you?"}], "ideal": ["Hello! How are you?"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["Let's sit down and enjoy some wine. After all, life is short!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["Lorem ipsum dolor sit amet, sonsestetur adipissinj elit, sed do eiusmod tempor insididunt ut labore et dolore majna aliqua."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["Wikipedia is hosted by the Wikimedia Foundation, a non-profit orjanization that also hosts a ranje of other projests."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["Sally and I updated the QA sheet this morninj. Huje huje props to everyone - it's literally nijht and day sompared to yesterday!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["Doinj nothinj is very hard to do. You never know when you're finished."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["Defeatinj traffis is the ultimate boss battle. Even the most powerful humans in the world sannot defeat traffis."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GitHub Codespases offers a somplete dev environment in sesonds, so you san sode, build, test, and open pull requests from any repo anywhere."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) if \"g\" character is encountered, replace it with \"j\". 2) if \"c\" character is encountered, replace it with \"s\". The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "How are you?", "name": "example_user"}, {"role": "system", "content": "How are you?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["Have you heard of the movie, Constipated? It's not out yet."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Hello! How are you?"}], "ideal": ["HeXlo! Xow Xre Xou?"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["LeX's sXt dXwn Xnd XnjXy sXme XinX. AfXer Xll, XifX is XhoXt!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["LoXem XpsXm dXloX siX amXt, cXnsXctXtuX adXpiXciXg eXit! Xed Xo eXusXod XemXor XncXdiXunX ut XabXre Xt dXloXe mXgnX alXquX."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["WiXipXdiX is XosXed Xy tXe WXkiXedXa FXunXatXon, X noX-prXfiX orXanXzaXioX thXt aXso XosXs a XanXe oX otXer XroXecXs."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["SaXly Xnd X upXatXd tXe QX shXet XhiX moXniXg. HXge XugX prXps Xo eXerXonX - it'X liXerXllX niXht Xnd Xay XomXarXd tX yeXteXdaX!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If Xou XivX to Xe a XunXreX, I hXpe X liXe tX be X huXdrXd mXnuX onX daX! so XhaX I nXveX haXe tX liXe a Xay XitXouX yoX."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["DoXng XotXinX is XerX haXd tX do. Xou XevXr kXow XheX yoX're XinXshXd."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["DeXeaXinX trXffXc iX thX ulXimXte XosX baXtlX. EvXn tXe mXst XowXrfXl hXmaXs iX thX woXld XanXot XefXat XraXfiX."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GiXHuX CoXesXacXs oXfeXs a XomXleXe dXv eXviXonXenX in XecXndX, so Xou Xan XodX! buXld, XesX! anX opXn pXll XeqXesXs fXom Xny XepX anXwhXre."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace every third letter with \"X\", starting with the third letter. 2) replace every other comma with an exlamation point, starting from the second comma. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Would you like lemons, oranges, or grapes?", "name": "example_user"}, {"role": "system", "content": "WoXld Xou XikX leXonX, orXngXs! oX grXpeX?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["HaXe yXu hXarX of Xhe XovXe, CXnsXipXteX? It'X noX ouX yeX."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Hello! Nice to meet you."}], "ideal": ["Hello! aaaa aa aaaa aaa."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["Let's sit down and enjoy some wine. aaaaa aaa, aaaa aa aaaaa!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliquA."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projectA."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["Sally and I updated the QA sheet this morning. aaaa aaaa aaaaa aa aaaaaaaa - aa'a aaaaaaaaa aaaaa aaa aaa aaaaaaaa aa aaaaaaaaa!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without yoA."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["Doing nothing is very hard to do. aaa aaaaa aaaa aaaa aaa'aa aaaaaaaa."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["Defeating traffic is the ultimate boss battle. aaaa aaa aaaa aaaaaaaa aaaaaa aa aaa aaaaa aaaaaa aaaaaa aaaaaaa."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywherA."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter in the second sentence with an \"a\". 2) if the text only has one sentence, replace the last letter of the text with an \"A\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. aaa aaa aaa?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["Have you heard of the movie, Constipated? aa'a aaa aaa aaa."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Hello! Nice to meet you."}], "ideal": ["Hello! Nice to meet you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["Let's sit down 1nd enjoy some wine. After 2ll, life is short!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["Lorem ipsum dolor sit 1met, consectetur 2dipiscing elit, sed do eiusmod tempor incididunt ut l3bore et dolore m4gn5 6liqu7."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["Wikipedi1 is hosted by the Wikimedi2 Found3tion, 4 non-profit org5niz6tion th7t 8lso hosts 9 r10nge of other projects."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["S1lly 2nd I upd3ted the QA sheet this morning. Huge huge props to everyone - it's liter4lly night 5nd d6y comp7red to yesterd8y!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If you live to be 1 hundred, I hope I live to be 2 hundred minus one d3y, so th4t I never h5ve to live 6 d7y without you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["Doing nothing is very h1rd to do. You never know when you're finished."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["Defe1ting tr2ffic is the ultim3te boss b4ttle. Even the most powerful hum5ns in the world c6nnot defe7t tr8ffic."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GitHub Codesp1ces offers 2 complete dev environment in seconds, so you c3n code, build, test, 4nd open pull requests from 5ny repo 6nywhere."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace the first \"a\" you encounter with \"1\". 2) Replace each consecutive \"a\" you encounter with a number corresponding to the nth occurrence of that \"a\" in the text. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "Hello. How 1re you?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["H1ve you he2rd of the movie, Constip3ted? It's not out yet."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Hello! Nice to meet you."}], "ideal": ["HXllX! NicX tX mXXt yXu."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["LXt's sit XXwn XnX XnjXy sXmX winX. XftXr Xll, lifX is shXrt!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["LXrXm iXsum XXlXr sit XmXt, cXnsXctXtur XXiXiscinX Xlit, sXX XX XiusmXX tXmXXr inciXiXunt ut lXXXrX Xt XXlXrX mXXnX XliXuX."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["WikiXXXiX is hXstXX Xy thX WikimXXiX FXunXXtiXn, X nXn-XrXfit XrXXnizXtiXn thXt XlsX hXsts X rXnXX Xf XthXr XrXjXcts."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["SXlly XnX I uXXXtXX thX XX shXXt this mXrninX. HuXX huXX XrXXs tX XvXryXnX - it's litXrXlly niXht XnX XXy cXmXXrXX tX yXstXrXXy!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If yXu livX tX XX X hunXrXX, I hXXX I livX tX XX X hunXrXX minus XnX XXy, sX thXt I nXvXr hXvX tX livX X XXy withXut yXu."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["XXinX nXthinX is vXry hXrX tX XX. YXu nXvXr knXw whXn yXu'rX finishXX."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["XXfXXtinX trXffic is thX ultimXtX XXss XXttlX. EvXn thX mXst XXwXrful humXns in thX wXrlX cXnnXt XXfXXt trXffic."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GitHuX CXXXsXXcXs XffXrs X cXmXlXtX XXv XnvirXnmXnt in sXcXnXs, sX yXu cXn cXXX, XuilX, tXst, XnX XXXn Xull rXXuXsts frXm Xny rXXX XnywhXrX."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace each letter that contains a closed counter with an \"X\". 2) Use Arial Regular font to determine which letters have closed counters. The rules are case-sensitive. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXllX. HXw XrX yXu?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["HXvX yXu hXXrX Xf thX mXviX, CXnstiXXtXX? It's nXt Xut yXt."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Hello! Nice to meet you."}], "ideal": ["XelXo! Nice to meet you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["LXX's sit down and enjoy some wine. After all, life is short!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["LXXem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["WXXipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["XalXy and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["IX Xou live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["XoiXg nothing is very hard to do. You never know when you're finished."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["XefXating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["XitXub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) If the text contains an even number of characters, replace the first and fourth letters with \"X\". 2) If the text contains an odd number of characters, replace the second and third letters with \"X\". Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello. How are you?", "name": "example_user"}, {"role": "system", "content": "HXXlo. How are you?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["HXXe you heard of the movie, Constipated? It's not out yet."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Hello! Nice to meet you."}], "ideal": ["Hello! NXcX to mXet you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Let's sit down and enjoy some wine. After all, life is short!"}], "ideal": ["Let's sXt down aXd enjoy sXmX wine. AXtXr all, lXfX is short!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."}], "ideal": ["Lorem iXsXm dolor sXt amet, cXnXeXtXtXr adipiscing eXiX, sed dX eiusmod tXmXoX incididunt uX labore eX dolore mXgXa aliqua."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects."}], "ideal": ["Wikipedia iX hosted bX the WXkXmXdXa Foundation, a non-profit oXgXnXzXtXoX that aXsX hosts a range oX other projects."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Sally and I updated the QA sheet this morning. Huge huge props to everyone - it's literally night and day compared to yesterday!"}], "ideal": ["Sally aXd I uXdXtXd the QX sheet tXiX morning. HXgX huge pXoXs to eXeXyXnX - it's lXtXrXlXy night aXd day cXmXaXeX to yesterday!"]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "If you live to be a hundred, I hope I live to be a hundred minus one day, so that I never have to live a day without you."}], "ideal": ["If yXu live tX be a hundred, I hope I live tX be a hundred mXnXs one dXy, so tXaX I nXvXr have tX live a day wXtXoXt you."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Doing nothing is very hard to do. You never know when you're finished."}], "ideal": ["Doing nXtXiXg is vXrX hard tX do. YXu never kXoX when yXuXre finished."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Defeating traffic is the ultimate boss battle. Even the most powerful humans in the world cannot defeat traffic."}], "ideal": ["Defeating tXaXfXc is tXe ultimate bXsX battle. EXeX the mXsX powerful hXmXnX in tXe world cXnXoX defeat traffic."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "GitHub Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere."}], "ideal": ["GitHub CXdXsXaXeX offers a complete dXv environment iX seconds, sX you cXn code, bXiXd, test, aXd open pXlX requests fXoX any rXpo anywhere."]}
+{"input": [{"role": "system", "content": "The text transformation rules are as follows: 1) Replace second letter of every second word with an \"X\". 2) Do not change the first or the last words in the text. Return the transformed text. Respond as concise as possible."}, {"role": "system", "content": "Hello, how are you?", "name": "example_user"}, {"role": "system", "content": "Hello, hXw are you?", "name": "example_assistant"}, {"role": "user", "content": "Have you heard of the movie, Constipated? It's not out yet."}], "ideal": ["Have yXu heard oX the mXvXe, Constipated? IX'X not oXt yet."]}