|
--- |
|
license: odc-by |
|
language: |
|
- en |
|
library_name: fasttext |
|
pipeline_tag: text-classification |
|
datasets: |
|
- HuggingFaceFW/fineweb-edu-llama3-annotations |
|
--- |
|
# FineWeb-Edu FastText classifier |
|
|
|
## Model summary |
|
This is a FastText classifier for judging the educational value of web pages based on training data [fineweb-edu-llama3-annotations](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-llama3-annotations). |
|
There are two objectives: |
|
- ⚡ throughput optimisation: It can classify more than 2000 examples per second in CPU, and so it can be used on-the-fly during pretraining/ to process huge data with CPU. |
|
- 🧪fasttext vs transformer based model: How does this lightweight model with limited capacity compare to the original model [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier)? |
|
|
|
|
|
The FastText approach is inspired by my independent development of educational classifier based on a different definition of educational value, which can be found at [kenhktsui/llm-data-textbook-quality-fasttext-classifier-v2](https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifier-v2). |
|
|
|
|
|
## 🛠️Usage |
|
```python |
|
from typing import List |
|
import re |
|
from huggingface_hub import hf_hub_download |
|
import fasttext |
|
|
|
|
|
model_hf = fasttext.load_model(hf_hub_download("kenhktsui/fineweb-edu-fasttext-classifier", "model.bin")) |
|
|
|
|
|
def replace_newlines(text: str) -> str: |
|
return re.sub("\n+", " ", text) |
|
|
|
|
|
def predict(text_list: List[str]) -> List[dict]: |
|
text_list = [replace_newlines(text) for text in text_list] |
|
pred = model_hf.predict(text_list) |
|
return [{"label": int(l[0].lstrip("__label__")), "score": s[0]} |
|
for l, s in zip(*pred)] |
|
|
|
|
|
predict(["Hi"]) |
|
# Output: [{'label': 0, 'score': 1.00001}] |
|
|
|
``` |
|
|
|
## 📊Evaluation |
|
The last 46867 samples are used as test data, but it's not the exact test data as in [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) |
|
### Classification Report |
|
``` |
|
precision recall f1-score support |
|
|
|
0 0.72 0.44 0.55 5704 |
|
1 0.73 0.87 0.80 26595 |
|
2 0.52 0.49 0.50 10350 |
|
3 0.48 0.33 0.39 3397 |
|
4 0.69 0.03 0.06 819 |
|
5 0.00 0.00 0.00 2 |
|
|
|
accuracy 0.68 46867 |
|
macro avg 0.52 0.36 0.38 46867 |
|
weighted avg 0.67 0.68 0.66 46867 |
|
``` |
|
|
|
The below table compares FastText model vs transformer based model. |
|
|
|
Label|This Model| [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) |
|
-----|-----|---- |
|
0|0.55 | 0.59 |
|
1|0.80 | 0.81 |
|
2|0.50 | 0.59 |
|
3|0.39 | 0.53 |
|
4|0.06 | 0.44 |
|
5|0.00 | 0.02 |
|
|
|
Label 0, 1, 2 are comparable to the original model. |
|
The performance degradation starts to be noticeable in label 3, and widen further in 4, which is due to limited capacity of fasttext model. |
|
So, this classifer can perform reasonably well in label 0, 1, 2, and also 3 with some degradation. |
|
|
|
### Confusion Matrix |
|
``` |
|
[ 2537 3098 65 4 0 0] |
|
[ 944 23037 2491 123 0 0] |
|
y_true [ 26 4742 5048 533 1 0] |
|
[ 4 434 1846 1105 8 0] |
|
[ 0 38 213 544 24 0] |
|
[ 0 0 0 0 2 0] |
|
y_pred |
|
``` |
|
|
|
The model has a accuracy of 68%, and it is more likely to underpredict educational value than overpredict so. The exhibited conservatism is good for filtering large amount of data. |
|
|
|
Predicted - Actual Rating | Frequency | % |
|
-----|-----|---- |
|
0|31751 | 67.7% |
|
-1|8078 | 17.2% |
|
+1| 6130 | 13.1% |
|
-2|673 | 1.4% |
|
+2|189 | 0.4% |
|
-3|42 | 0.1% |
|
+3|4 | 0.0% |
|
|
|
|
|
### Alignment with [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) |
|
Spearman rank-order correlation coefficient is 0.5881 in MiniPile train split and 0.5832 in test split, indicating a moderately strong monotonic relationship in over 1 million representative document in web data. |
|
|
