metadata

language:
  - en
license: mit
tags:
  - text-classfication
  - int8
  - PostTrainingDynamic
datasets:
  - glue
metrics:
  - f1
model-index:
  - name: camembert-base-mrpc-int8-dynamic
    results:
      - task:
          name: Text Classification
          type: text-classification
        dataset:
          name: GLUE MRPC
          type: glue
          args: mrpc
        metrics:
          - name: F1
            type: f1
            value: 0.8842832469775476

INT8 camembert-base-mrpc

Post-training dynamic quantization

This is an INT8 PyTorch model quantized with Intel® Neural Compressor.

The original fp32 model comes from the fine-tuned model camembert-base-mrpc.

The linear module roberta.encoder.layer.6.attention.self.query falls back to fp32 to meet the 1% relative accuracy loss.

Test result

Batch size = 8
Amazon Web Services c6i.xlarge (Intel ICE Lake: 4 vCPUs, 8g Memory) instance.

	INT8	FP32
Throughput (samples/sec)	24.745	13.078
Accuracy (eval-f1)	0.8843	0.8928
Model size (MB)	180	422

Load with Intel® Neural Compressor (build from source):

from neural_compressor.utils.load_huggingface import OptimizedModel
int8_model = OptimizedModel.from_pretrained(
    'Intel/camembert-base-mrpc-int8-dynamic',
)

Notes:

The INT8 model has better performance than the FP32 model when the CPU is fully occupied. Otherwise, there will be the illusion that INT8 is inferior to FP32.