CF_CodeTesting.yaml

description: "ToDo: add description"

input_data_transformations: []
input_keys:
  - "code"
  - "public_tests_individual_io"

output_keys:
  - "all_tests_passed"
  - "testing_results_summary"
output_data_transformations:
  - _target_: martinjosifoski.CC_flows.src.data_transformations.CorrectnessFlag  # ToDo: This import style would not work if the flow is synced in the current implementation (the outer directory is a hash and not the name; figure out how to do the import robustly; e.g., using relative imports)
    input_key: "raw_response.public_tests_results"
    output_key: "all_tests_passed"
  - _target_: martinjosifoski.CC_flows.src.data_transformations.TestingResultsSummaryGeneration
    output_key: "testing_results_summary"

    single_test_error_message: True

    no_error_template: |2-
      ${.issue_title}
      All of the executed tests passed.

    compilation_error_template: |2-
      ${.issue_title}
      The execution resulted in a compilation error.
      ## Compilation error message:
      {{error_message}}
    timeout_error_template: |2-
      ${.issue_title}
      The execution timed out, the solution is not efficient enough.
    runtime_error_template: |2-
      ${.issue_title}
      The execution resulted in a runtime error on the following test.
      ## [Failed test] Input
      ```
      {{test_input}}
      ```
      ## [Failed test] Runtime error message
      {{error_message}}
    single_test_error_template: |2-
      ${.issue_title}
      The Python code does not solve the problem in the problem description due to logical errors. It fails the following test:
      ## [Failed test] Input
      ```
      {{test_input}}
      ```
      ## [Failed test] Expected output
      ```
      {{expected_output}}
      ```
      ## [Failed test] Generated output
      ```
      {{generated_output}}
      ```
    all_tests_header: |2-
      ${.issue_title}
      The Python code does not solve the problem in the problem description due to logical errors. It fails on the following tests.
    test_error_template: |2-
      ## [Failed test {{idx}}]
      ### [Failed test {{idx}}] Input
      ```
      {{test_input}}
      ```
      ### [Failed test {{idx}}] Expected output
      ```
      {{expected_output}}
      ```
      ### [Failed test {{idx}}] Generated output
      ```
      {{generated_output}}
      ```
    tests_separator: "\n\n"

    issue_title: "# Issue with the last proposed solution"

    feedback_title: "# Feedback on the last proposed solution"

    no_code_template: |2-
      ${.feedback_title}
      The code was not provided in the correct output format specified in the request or it was not provided at all.
    feedback_only_template: |2-
      ${.feedback_title}
      {{feedback_content}}
    feedback_and_issue_template: |2-
      {{issue_description}}

      {{feedback_content}}