---
license: apache-2.0
base_model: distilbert-base-uncased
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: distilbert-base-uncased-finetuned-clinc
  results: []
datasets:
- clinc_oos
library_name: transformers
pipeline_tag: text-classification
---

# Transformer Efficiency and Knowledge Distillation

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.7872
- Accuracy: 0.9206

## Model description

This setup involves benchmarking the performance of a fine-tuned BERT model (transformersbook/bert-base-uncased-finetuned-clinc) and applying knowledge distillation to train a smaller DistilBERT model. The BERT model is used for text classification tasks, and its efficiency is evaluated in terms of accuracy, model size, and latency. The DistilBERT model is trained to mimic the BERT model's performance while being more efficient.

## Intended uses & limitations

### Intended uses:

Evaluating the performance efficiency of transformer models.
Applying knowledge distillation to create smaller and faster models for text classification.

### Limitations:

The benchmark results are specific to the dataset used (CLINC150) and may not generalize to other datasets.
Knowledge distillation relies on the quality and performance of the teacher model.

## Training and evaluation data

The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits.

## Training procedure

### Training and evaluation data

The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits.

### Performance Benchmark

The performance of the BERT model is evaluated using the PerformanceBenchmark class, which measures accuracy, model size, and latency.

### Accuracy

The model's accuracy is computed on the test set of the CLINC150 dataset.
accuracy_score = load_metric("accuracy") 

### Model Size

The size of the model is computed by saving its state dictionary to disk and measuring the file size in megabytes.

def compute_size(self):
    state_dict = self.pipeline.model.state_dict()
    tmp_path = Path("model.pt")
    torch.save(state_dict, tmp_path)
    size_mb = Path(tmp_path).stat().st_size / (1024 * 1024)
    tmp_path.unlink()
    return {"size_mb": size_mb}
    
### Latency

The average latency per query is measured over a sample of 100 queries.

def time_pipeline(self):
    latencies = []
    for example in self.dataset[:100]:
        start_time = perf_counter()
        _ = self.pipeline(example)
        latency = perf_counter() - start_time
        latencies.append(latency)
    time_avg_ms = 1000 * np.mean(latencies)
    time_std_ms = 1000 * np.std(latencies)
    return {"time_avg_ms": time_avg_ms, "time_std_ms": time_std_ms}
    
### Knowledge Distillation

Knowledge distillation is used to train a smaller DistilBERT model using the predictions of the fine-tuned BERT model as soft labels.

### Distillation Process

Teacher Model: transformersbook/bert-base-uncased-finetuned-clinc
Student Model: distilbert-base-uncased
The distillation process involves computing a weighted average of the cross-entropy loss with the ground truth labels and the Kullback-Leibler divergence between the teacher and student model predictions.

class DistillationTrainer(Trainer):
  def compute_loss(self, model, inputs, return_outputs=False):
    outputs_stu = model(**inputs)
    loss_ce = outputs_stu.loss
    logits_stu = outputs_stu.logits
    with torch.no_grad():
      outputs_tea = self.teacher(**inputs)
      logits_tea = outputs_tea.logits
    loss_fct = nn.KLDivLoss(reduction="batchmean")
    loss_kd = self.args.temperature ** 2 * loss_fct(
        F.log_softmax(logits_stu / self.args.temperature, dim=-1),
        F.softmax(logits_tea / self.args.temperature, dim=-1)
    )
    loss = self.args.alpha * loss_ce + (1. - self.args.alpha) * loss_kd
    return (loss, outputs_stu) if return_outputs else loss
    
### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 48
- eval_batch_size: 48
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| No log        | 1.0   | 318  | 3.2931          | 0.7255   |
| 3.8009        | 2.0   | 636  | 1.8849          | 0.8526   |
| 3.8009        | 3.0   | 954  | 1.1702          | 0.8897   |
| 1.7128        | 4.0   | 1272 | 0.8717          | 0.9145   |
| 0.9206        | 5.0   | 1590 | 0.7872          | 0.9206   |


### Framework versions

- Transformers 4.41.1
- Pytorch 2.3.0+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1