Update README.md

README.md

@@ -37,11 +37,6 @@ This model is ready for commercial use.
 # License
 This model is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
 
-# References
-* [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://arxiv.org/abs/2111.09543)
-* [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://github.com/microsoft/DeBERTa)
-* [Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155)
-
 # Model Architecture
 The model architecture uses a DeBERTa backbone and incorporates multiple classification heads, each dedicated to a task categorization or complexity dimension. This approach enables the training of a unified network, allowing it to predict simultaneously during inference. Deberta-v3-base can theoretically handle up to 12k tokens, but default context length is set at 512 tokens.
 
@@ -50,6 +45,77 @@ The model architecture uses a DeBERTa backbone and incorporates multiple classif
 
 The inference code for this model is available through the NeMo Curator GitHub repository. Check out this [example notebook](https://github.com/NVIDIA/NeMo-Curator/tree/main/tutorials/distributed_data_classification) to get started.
 
+# Input & Output
+## Input
+* Input Type: Text
+* Input Format: String
+* Input Parameters: 1D
+* Other Properties Related to Input: Token Limit of 512 tokens
+
+## Output
+* Output Type: Text/Numeric Classifications
+* Output Format: String & Numeric
+* Output Parameters: 1D
+* Other Properties Related to Output: None
+
+## Examples
+
+```
+Prompt: Write a mystery set in a small town where an everyday object goes missing, causing a ripple of curiosity and suspicion. Follow the investigation and reveal the surprising truth behind the disappearance.
+```
+
+| Task            | Complexity | Creativity | Reasoning | Contextual Knowledge | Domain Knowledge | Constraints | # of Few Shots |
+|------------------|------------|------------|-----------|-----------------------|------------------|-------------|----------------|
+| Text Generation | 0.472      | 0.867      | 0.056     | 0.048                | 0.226            | 0.785       | 0              |
+
+```
+Prompt: Antibiotics are a type of medication used to treat bacterial infections. They work by either killing the bacteria or preventing them from reproducing, allowing the body’s immune system to fight off the infection. Antibiotics are usually taken orally in the form of pills, capsules, or liquid solutions, or sometimes administered intravenously. They are not effective against viral infections, and using them inappropriately can lead to antibiotic resistance. Explain the above in one sentence.
+```
+
+| Task           | Complexity | Creativity | Reasoning | Contextual Knowledge | Domain Knowledge | Constraints | # of Few Shots |
+|-----------------|------------|------------|-----------|-----------------------|------------------|-------------|----------------|
+| Summarization  | 0.133      | 0.003      | 0.014     | 0.003                | 0.644            | 0.211       | 0              |
+
+# Software Integration
+* Runtime Engine: Python 3.10 and NeMo Curator
+* Supported Hardware Microarchitecture Compatibility: NVIDIA GPU, Volta™ or higher (compute capability 7.0+), CUDA 12 (or above)
+* Preferred/Supported Operating System(s): Ubuntu 22.04/20.04
+
+# Model Version
+Prompt Task and Complexity Classifier v1.1
+
+# Training, Testing, and Evaluation Datasets
+## Training Data
+* 4024 English prompts with task distribution outlined below
+* Prompts were annotated by humans according to task and complexity taxonomies
+
+Task distribution:
+| Task            | Count |
+|------------------|-------|
+| Open QA         | 1214  |
+| Closed QA       | 786   |
+| Text Generation | 480   |
+| Chatbot         | 448   |
+| Classification  | 267   |
+| Summarization   | 230   |
+| Code Generation | 185   |
+| Rewrite         | 169   |
+| Other           | 104   |
+| Brainstorming   | 81    |
+| Extraction      | 60    |
+| Total       | 4024  |
+
+## Evaluation
+For evaluation, Top-1 accuracy metric was used, which involves matching the category with the highest probability to the expected answer. Additionally, n-fold cross-validation was used to produce n different values for this metric to verify the consistency of the results. The table below displays the average of the top-1 accuracy values for the N folds calculated for each complexity dimension separately.
+
+| | Task Accuracy    | Creative Accuracy | Reasoning Accuracy | Contextual Accuracy | FewShots Accuracy | Domain Accuracy | Constraint Accuracy |
+|-|------------------|-------------------|--------------------|---------------------|-------------------|-----------------|---------------------|
+| Average of 10 Folds | 0.981            | 0.996            | 0.997             | 0.981              | 0.979            | 0.937          | 0.991 |
+
+# Inference
+* Engine: PyTorch
+* Test Hardware: A10G
+
 # How to Use in Transformers
 To use the prompt task and complexity classifier, use the following code:
 
@@ -263,76 +329,10 @@ print(result)
 # {'task_type_1': ['Code Generation'], 'task_type_2': ['Text Generation'], 'task_type_prob': [0.767], 'creativity_scope': [0.0826], 'reasoning': [0.0632], 'contextual_knowledge': [0.056], 'number_of_few_shots': [0], 'domain_knowledge': [0.9803], 'no_label_reason': [0.0], 'constraint_ct': [0.5578], 'prompt_complexity_score': [0.27822]}
 ```
 
-# Input & Output
-## Input
-* Input Type: Text
-* Input Format: String
-* Input Parameters: 1D
-* Other Properties Related to Input: Token Limit of 512 tokens
-
-## Output
-* Output Type: Text/Numeric Classifications
-* Output Format: String & Numeric
-* Output Parameters: 1D
-* Other Properties Related to Output: None
-
-## Examples
-
-```
-Prompt: Write a mystery set in a small town where an everyday object goes missing, causing a ripple of curiosity and suspicion. Follow the investigation and reveal the surprising truth behind the disappearance.
-```
-
-| Task            | Complexity | Creativity | Reasoning | Contextual Knowledge | Domain Knowledge | Constraints | # of Few Shots |
-|------------------|------------|------------|-----------|-----------------------|------------------|-------------|----------------|
-| Text Generation | 0.472      | 0.867      | 0.056     | 0.048                | 0.226            | 0.785       | 0              |
-
-```
-Prompt: Antibiotics are a type of medication used to treat bacterial infections. They work by either killing the bacteria or preventing them from reproducing, allowing the body’s immune system to fight off the infection. Antibiotics are usually taken orally in the form of pills, capsules, or liquid solutions, or sometimes administered intravenously. They are not effective against viral infections, and using them inappropriately can lead to antibiotic resistance. Explain the above in one sentence.
-```
-
-| Task           | Complexity | Creativity | Reasoning | Contextual Knowledge | Domain Knowledge | Constraints | # of Few Shots |
-|-----------------|------------|------------|-----------|-----------------------|------------------|-------------|----------------|
-| Summarization  | 0.133      | 0.003      | 0.014     | 0.003                | 0.644            | 0.211       | 0              |
-
-# Software Integration
-* Runtime Engine: Python 3.10 and NeMo Curator
-* Supported Hardware Microarchitecture Compatibility: NVIDIA GPU, Volta™ or higher (compute capability 7.0+), CUDA 12 (or above)
-* Preferred/Supported Operating System(s): Ubuntu 22.04/20.04
-
-# Model Version
-Prompt Task and Complexity Classifier v1.1
-
-# Training, Testing, and Evaluation Datasets
-## Training Data
-* 4024 English prompts with task distribution outlined below
-* Prompts were annotated by humans according to task and complexity taxonomies
-
-Task distribution:
-| Task            | Count |
-|------------------|-------|
-| Open QA         | 1214  |
-| Closed QA       | 786   |
-| Text Generation | 480   |
-| Chatbot         | 448   |
-| Classification  | 267   |
-| Summarization   | 230   |
-| Code Generation | 185   |
-| Rewrite         | 169   |
-| Other           | 104   |
-| Brainstorming   | 81    |
-| Extraction      | 60    |
-| Total       | 4024  |
-
-## Evaluation
-For evaluation, Top-1 accuracy metric was used, which involves matching the category with the highest probability to the expected answer. Additionally, n-fold cross-validation was used to produce n different values for this metric to verify the consistency of the results. The table below displays the average of the top-1 accuracy values for the N folds calculated for each complexity dimension separately.
-
-| | Task Accuracy    | Creative Accuracy | Reasoning Accuracy | Contextual Accuracy | FewShots Accuracy | Domain Accuracy | Constraint Accuracy |
-|-|------------------|-------------------|--------------------|---------------------|-------------------|-----------------|---------------------|
-| Average of 10 Folds | 0.981            | 0.996            | 0.997             | 0.981              | 0.979            | 0.937          | 0.991 |
-
-# Inference
-* Engine: PyTorch
-* Test Hardware: A10G
+# References
+* [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://arxiv.org/abs/2111.09543)
+* [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://github.com/microsoft/DeBERTa)
+* [Training language models to follow instructions with human feedback](https://arxiv.org/pdf/2203.02155)
 
 # Ethical Considerations
 NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.