Adriana213/distilbert-base-uncased-finetuned-clinc

Transformer Efficiency and Knowledge Distillation

This model is a fine-tuned version of distilbert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7872
• Accuracy: 0.9206

Model description

This setup involves benchmarking the performance of a fine-tuned BERT model (transformersbook/bert-base-uncased-finetuned-clinc) and applying knowledge distillation to train a smaller DistilBERT model. The BERT model is used for text classification tasks, and its efficiency is evaluated in terms of accuracy, model size, and latency. The DistilBERT model is trained to mimic the BERT model's performance while being more efficient.

Intended uses & limitations

Intended uses:

Evaluating the performance efficiency of transformer models. Applying knowledge distillation to create smaller and faster models for text classification.

Limitations:

The benchmark results are specific to the dataset used (CLINC150) and may not generalize to other datasets. Knowledge distillation relies on the quality and performance of the teacher model.

Training and evaluation data

The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits.

Training procedure

Training and evaluation data

The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits.

Performance Benchmark

The performance of the BERT model is evaluated using the PerformanceBenchmark class, which measures accuracy, model size, and latency.

Accuracy

The model's accuracy is computed on the test set of the CLINC150 dataset. accuracy_score = load_metric("accuracy")

Model Size

The size of the model is computed by saving its state dictionary to disk and measuring the file size in megabytes.

def compute_size(self): state_dict = self.pipeline.model.state_dict() tmp_path = Path("model.pt") torch.save(state_dict, tmp_path) size_mb = Path(tmp_path).stat().st_size / (1024 * 1024) tmp_path.unlink() return {"size_mb": size_mb}

Latency

The average latency per query is measured over a sample of 100 queries.

def time_pipeline(self): latencies = [] for example in self.dataset[:100]: start_time = perf_counter() _ = self.pipeline(example) latency = perf_counter() - start_time latencies.append(latency) time_avg_ms = 1000 * np.mean(latencies) time_std_ms = 1000 * np.std(latencies) return {"time_avg_ms": time_avg_ms, "time_std_ms": time_std_ms}

Knowledge Distillation

Knowledge distillation is used to train a smaller DistilBERT model using the predictions of the fine-tuned BERT model as soft labels.

Distillation Process

Teacher Model: transformersbook/bert-base-uncased-finetuned-clinc Student Model: distilbert-base-uncased The distillation process involves computing a weighted average of the cross-entropy loss with the ground truth labels and the Kullback-Leibler divergence between the teacher and student model predictions.

class DistillationTrainer(Trainer): def compute_loss(self, model, inputs, return_outputs=False): outputs_stu = model(**inputs) loss_ce = outputs_stu.loss logits_stu = outputs_stu.logits with torch.no_grad(): outputs_tea = self.teacher(**inputs) logits_tea = outputs_tea.logits loss_fct = nn.KLDivLoss(reduction="batchmean") loss_kd = self.args.temperature ** 2 * loss_fct( F.log_softmax(logits_stu / self.args.temperature, dim=-1), F.softmax(logits_tea / self.args.temperature, dim=-1) ) loss = self.args.alpha * loss_ce + (1. - self.args.alpha) * loss_kd return (loss, outputs_stu) if return_outputs else loss

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 48
• eval_batch_size: 48
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	1.0	318	3.2931	0.7255
3.8009	2.0	636	1.8849	0.8526
3.8009	3.0	954	1.1702	0.8897
1.7128	4.0	1272	0.8717	0.9145
0.9206	5.0	1590	0.7872	0.9206

Framework versions

• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1