movadek/uk-imm-court-outcome

UK Immigration Rulings Outcome Classifier

This deberta-v3-large model is fine-tuned to classify the outcomes of the rulings of the UK Upper Tribunal (Immigration and Asylum Chamber) using hand-coded text data from this court.

The model predicts whether an input text sequence indicates appeal success, failure or neither.

Out-of-sample final test metrics:

accuracy: 0.9289

matthews_correlation: 0.8927

We have not tested how well the classifier generalizes to any other court.

Fine-tuning parameters

The base model is deberta-v3-large. It was fine-tuned using the following parameters on a single A100 80GB GPU:

learning_rate=2e-5,

per_device_train_batch_size=16,

per_device_eval_batch_size=16,

num_train_epochs=8,

weight_decay=0.01,

evaluation_strategy="epoch"

Data

The training data comes from the website of the UK Upper Tribunal (Immigration and Asylum Chamber): https://tribunalsdecisions.service.gov.uk/utiac/

Court decisions were first split into paragraphs. In the second step, paragraphs likely containing statement of outcome were identified using regular expressions in an over-inclusive manner.

Paragraphs (split if longer than 700 characters) were subsequently hand-coded by a single researcher for indicating appeal success, failure or neither. Appeal success means that the Upper Tribunal sets aside (in full or in part) the decision of the First-tier Tribunal (FtT). Appeal failure means that the Upper Tribunal upholds the FtT decision. neither indicates an irrelevant or inconclusive paragraph.

For example, the following paragraph is coded as success:

44. The determination of the First-tier Tribunal is set aside. The following decision is substituted: the appeal, as originally brought to the FtT, is allowed.

The hand-coded data was split 80-10-10 into a training, evaluation and test set.

Training set size: 3929

Evaluation set size: 491

Test set size: 492