Van Tuan DANG
commited on
Commit
·
a2d8783
1
Parent(s):
4afd7f7
feat: Add Readmel
Browse files
README.md
ADDED
@@ -0,0 +1,97 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
pipeline_tag: sentence-similarity
|
3 |
+
language: fr
|
4 |
+
datasets:
|
5 |
+
- stsb_multi_mt
|
6 |
+
tags:
|
7 |
+
- Text
|
8 |
+
- Sentence Similarity
|
9 |
+
- Sentence-Embedding
|
10 |
+
- camembert-base
|
11 |
+
license: apache-2.0
|
12 |
+
model-index:
|
13 |
+
- name: sentence-camembert-base by Van Tuan DANG
|
14 |
+
results:
|
15 |
+
- task:
|
16 |
+
name: Sentence-Embedding
|
17 |
+
type: Text Similarity
|
18 |
+
dataset:
|
19 |
+
name: Text Similarity fr
|
20 |
+
type: stsb_multi_mt
|
21 |
+
args: fr
|
22 |
+
metrics:
|
23 |
+
- name: Test Pearson correlation coefficient
|
24 |
+
type: Pearson_correlation_coefficient
|
25 |
+
value: xx.xx
|
26 |
+
---
|
27 |
+
|
28 |
+
## {Model}
|
29 |
+
|
30 |
+
Cross-Encoder for sentence-similarity
|
31 |
+
|
32 |
+
This model was trained using [sentence-transformers](https://www.SBERT.net) Cross-Encoder class.
|
33 |
+
|
34 |
+
## Training Data
|
35 |
+
This model was trained on the [STS benchmark dataset](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.
|
36 |
+
|
37 |
+
|
38 |
+
## Usage (Sentence-Transformers)
|
39 |
+
|
40 |
+
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
41 |
+
|
42 |
+
```
|
43 |
+
pip install -U sentence-transformers
|
44 |
+
```
|
45 |
+
|
46 |
+
Then you can use the model like this:
|
47 |
+
|
48 |
+
```python
|
49 |
+
from sentence_transformers import CrossEncoder
|
50 |
+
model = CrossEncoder('model_name', max_length=128)
|
51 |
+
scores = model.predict([('Un avion est en train de décoller.', "Un homme joue d'une grande flûte."), ("Un homme étale du fromage râpé sur une pizza.", "Une personne jette un chat au plafond") ])
|
52 |
+
|
53 |
+
```
|
54 |
+
## Evaluation
|
55 |
+
The model can be evaluated as follows on the French test data of stsb.
|
56 |
+
```python
|
57 |
+
from sentence_transformers import SentenceTransformer
|
58 |
+
from sentence_transformers.readers import InputExample
|
59 |
+
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator
|
60 |
+
from datasets import load_dataset
|
61 |
+
def convert_dataset(dataset):
|
62 |
+
dataset_samples=[]
|
63 |
+
for df in dataset:
|
64 |
+
score = float(df['similarity_score'])/5.0 # Normalize score to range 0 ... 1
|
65 |
+
inp_example = InputExample(texts=[df['sentence1'],
|
66 |
+
df['sentence2']], label=score)
|
67 |
+
dataset_samples.append(inp_example)
|
68 |
+
return dataset_samples
|
69 |
+
|
70 |
+
# Loading the dataset for evaluation
|
71 |
+
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")
|
72 |
+
df_test = load_dataset("stsb_multi_mt", name="fr", split="test")
|
73 |
+
|
74 |
+
# Convert the dataset for evaluation
|
75 |
+
|
76 |
+
# For Dev set:
|
77 |
+
dev_samples = convert_dataset(df_dev)
|
78 |
+
val_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name='sts-dev')
|
79 |
+
val_evaluator(model, output_path="./")
|
80 |
+
|
81 |
+
# For Test set
|
82 |
+
|
83 |
+
test_samples = convert_dataset(df_test)
|
84 |
+
test_evaluator = CECorrelationEvaluator.from_input_examples(test_samples, name='sts-test')
|
85 |
+
test_evaluator(models, output_path="./")
|
86 |
+
```
|
87 |
+
**Test Result**:
|
88 |
+
The performance is measured using Pearson and Spearman correlation:
|
89 |
+
- On dev
|
90 |
+
| Model | Pearson correlation | Spearman correlation | #params |
|
91 |
+
| ------------- | ------------- | ------------- |------------- |
|
92 |
+
| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 90.11 |90.01 | 336M |
|
93 |
+
|
94 |
+
- On test
|
95 |
+
| Model | Pearson correlation | Spearman correlation |
|
96 |
+
| ------------- | ------------- | ------------- |
|
97 |
+
| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 88.16 | 87.57|
|