Commit
•
5acf869
1
Parent(s):
676f964
Update README.md
Browse files
README.md
CHANGED
@@ -1,8 +1,139 @@
|
|
1 |
---
|
2 |
pipeline_tag: sentence-similarity
|
|
|
|
|
|
|
3 |
tags:
|
4 |
-
-
|
5 |
-
-
|
6 |
-
-
|
7 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
8 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
pipeline_tag: sentence-similarity
|
3 |
+
language: fr
|
4 |
+
datasets:
|
5 |
+
- stsb_multi_mt
|
6 |
tags:
|
7 |
+
- Text
|
8 |
+
- Sentence Similarity
|
9 |
+
- Sentence-Embedding
|
10 |
+
- camembert-large
|
11 |
+
license: apache-2.0
|
12 |
+
model-index:
|
13 |
+
- name: sentence-camembert-large by Van Tuan DANG
|
14 |
+
results:
|
15 |
+
- task:
|
16 |
+
name: Sentence-Embedding
|
17 |
+
type: Text Similarity
|
18 |
+
dataset:
|
19 |
+
name: Text Similarity fr
|
20 |
+
type: stsb_multi_mt
|
21 |
+
args: fr
|
22 |
+
metrics:
|
23 |
+
- name: Test Pearson correlation coefficient
|
24 |
+
type: Pearson_correlation_coefficient
|
25 |
+
value: 88.63
|
26 |
+
---
|
27 |
+
## Description:
|
28 |
+
This [**Sentence-CamemBERT-Large**] Model is an Embedding Model for French developed by [La Javaness](https://www.lajavaness.com/). The purpose of this embedding model is to represent the content and semantics of a French sentence as a mathematical vector, allowing it to understand the meaning of the text beyond individual words in queries and documents. It offers powerful semantic search capabilities.
|
29 |
+
## Pre-trained sentence embedding models are state-of-the-art of Sentence Embeddings for French.
|
30 |
+
|
31 |
+
The [Lajavaness/sentence-camembert-large](https://huggingface.co/Lajavaness/sentence-camembert-large) model is an improvement over the [dangvantuan/sentence-camembert-base](https://huggingface.co/dangvantuan/sentence-camembert-large) offering greater robustness and better performance on all STS benchmark datasets. It has been fine-tuned using the pre-trained [facebook/camembert-large](https://huggingface.co/camembert/camembert-large) and
|
32 |
+
[Siamese BERT-Networks with 'sentences-transformers'](https://www.sbert.net/) on dataset [stsb](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train). Additionally, it has been combined with [Augmented SBERT](https://aclanthology.org/2021.naacl-main.28.pdf) on dataset [stsb](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train). The model benefits from Pair Sampling Strategies using two models: [CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) and [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)
|
33 |
+
|
34 |
+
## Usage
|
35 |
+
The model can be used directly (without a language model) as follows:
|
36 |
+
|
37 |
+
```python
|
38 |
+
from sentence_transformers import SentenceTransformer
|
39 |
+
model = SentenceTransformer("Lajavaness/sentence-camembert-large")
|
40 |
+
|
41 |
+
sentences = ["Un avion est en train de décoller.",
|
42 |
+
"Un homme joue d'une grande flûte.",
|
43 |
+
"Un homme étale du fromage râpé sur une pizza.",
|
44 |
+
"Une personne jette un chat au plafond.",
|
45 |
+
"Une personne est en train de plier un morceau de papier.",
|
46 |
+
]
|
47 |
+
|
48 |
+
embeddings = model.encode(sentences)
|
49 |
+
```
|
50 |
+
|
51 |
+
## Evaluation
|
52 |
+
The model can be evaluated as follows on the French test data of stsb.
|
53 |
+
|
54 |
+
```python
|
55 |
+
from sentence_transformers import SentenceTransformer
|
56 |
+
from sentence_transformers.readers import InputExample
|
57 |
+
from datasets import load_dataset
|
58 |
+
def convert_dataset(dataset):
|
59 |
+
dataset_samples=[]
|
60 |
+
for df in dataset:
|
61 |
+
score = float(df['similarity_score'])/5.0 # Normalize score to range 0 ... 1
|
62 |
+
inp_example = InputExample(texts=[df['sentence1'],
|
63 |
+
df['sentence2']], label=score)
|
64 |
+
dataset_samples.append(inp_example)
|
65 |
+
return dataset_samples
|
66 |
+
|
67 |
+
# Loading the dataset for evaluation
|
68 |
+
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")
|
69 |
+
df_test = load_dataset("stsb_multi_mt", name="fr", split="test")
|
70 |
+
|
71 |
+
# Convert the dataset for evaluation
|
72 |
+
|
73 |
+
# For Dev set:
|
74 |
+
dev_samples = convert_dataset(df_dev)
|
75 |
+
val_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(dev_samples, name='sts-dev')
|
76 |
+
val_evaluator(model, output_path="./")
|
77 |
+
|
78 |
+
# For Test set:
|
79 |
+
test_samples = convert_dataset(df_test)
|
80 |
+
test_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(test_samples, name='sts-test')
|
81 |
+
test_evaluator(model, output_path="./")
|
82 |
+
```
|
83 |
+
|
84 |
+
**Test Result**:
|
85 |
+
The performance is measured using Pearson and Spearman correlation:
|
86 |
+
- On dev
|
87 |
+
|
88 |
+
|
89 |
+
| Model | Pearson correlation | Spearman correlation | #params |
|
90 |
+
| ------------- | ------------- | ------------- |------------- |
|
91 |
+
| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)| **88.63** |**88.46** | 336M|
|
92 |
+
| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)| 88.2 |88.02 | 336M|
|
93 |
+
| [Sahajtomar/french_semanti](https://huggingface.co/Sahajtomar/french_semantic)| 87.44 |87.30 | 336M|
|
94 |
+
| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base)| 87.14 |87.10 | 137M |
|
95 |
+
| [GPT-3 (text-davinci-003)](https://platform.openai.com/docs/models) | 85 | NaN|175B |
|
96 |
+
| [GPT-(text-embedding-ada-002)](https://platform.openai.com/docs/models) | 79.75 | 80.44|NaN |
|
97 |
+
|
98 |
+
|
99 |
+
- On test, Pearson and Spearman correlation are evaluated on many different benchmark datasets:
|
100 |
+
|
101 |
+
|
102 |
+
**Pearson score**
|
103 |
+
| Model | [STS-B](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train) | [STS12-fr ](https://huggingface.co/datasets/Lajavaness/STS12-fr)| [STS13-fr](https://huggingface.co/datasets/Lajavaness/STS13-fr) | [STS14-fr](https://huggingface.co/datasets/Lajavaness/STS14-fr) | [STS15-fr](https://huggingface.co/datasets/Lajavaness/STS15-fr) | [STS16-fr](https://huggingface.co/datasets/Lajavaness/STS16-fr) | [SICK-fr](https://huggingface.co/datasets/Lajavaness/SICK-fr) | params |
|
104 |
+
|------------------------------------------|-------|----------|----------|----------|----------|----------|---------|--------|
|
105 |
+
| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | **86.26** | **87.42** | **89.34** | **88.05** | **88.91** | 77.15 | 83.13 | 336M |
|
106 |
+
| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | 85.88 | 87.28 | 89.25 | 87.91 | 88.54 | 76.90 | 83.26 | 336M |
|
107 |
+
| [Sahajtomar/french_semantic](https://huggingface.co/Sahajtomar/french_semantic) | 85.80 | 86.05 | 88.50 | 86.57 | 87.49 | 77.85 | 83.27 | 336 |
|
108 |
+
| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base) | 85.39 | 86.64 | 87.24 | 85.68 | 87.99 | 75.78 | 82.84 | 137M |
|
109 |
+
| [GPT3 (text-embedding-ada-002)](https://platform.openai.com/docs/models) | 79.03 | 66.16 | 75.48 | 70.69 | 77.88 | 65.18 | - | - |
|
110 |
+
|
111 |
+
|
112 |
+
**Spearman score**
|
113 |
+
| Model | [STS-B](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train) | [STS12-fr ](https://huggingface.co/datasets/Lajavaness/STS12-fr)| [STS13-fr](https://huggingface.co/datasets/Lajavaness/STS13-fr) | [STS14-fr](https://huggingface.co/datasets/Lajavaness/STS14-fr) | [STS15-fr](https://huggingface.co/datasets/Lajavaness/STS15-fr) | [STS16-fr](https://huggingface.co/datasets/Lajavaness/STS16-fr) | [SICK-fr](https://huggingface.co/datasets/Lajavaness/SICK-fr) | params |
|
114 |
+
|:-------------------------------------|-------:|---------:|---------:|---------:|---------:|---------:|--------:|:-------|
|
115 |
+
| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | **86.14** | **81.22** | 88.61 | **86.28** | **89.01** | 78.65 | **77.71** | 336M |
|
116 |
+
| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | 85.78 | 81.09 | 88.68 | 85.81 | 88.56 | 78.49 | 77.70 | 336M |
|
117 |
+
| [Sahajtomar/french_semantic](https://huggingface.co/Sahajtomar/french_semantic) | 85.55 | 77.92 | 87.85 | 83.96 | 87.63 | 79.07 | 77.14 | 137M |
|
118 |
+
| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base) | 85.67 | 79.97 | 86.91 | 84.57 | 88.10 | 77.84 | 77.55 | 336 |
|
119 |
+
| [GPT3 (text-embedding-ada-002)](https://platform.openai.com/docs/models) | 77.53 | 64.27 | 76.41 | 69.63 | 78.65 | 75.30 | - | - |
|
120 |
+
|
121 |
+
|
122 |
+
|
123 |
+
## Citation
|
124 |
+
|
125 |
+
|
126 |
+
@article{reimers2019sentence,
|
127 |
+
title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
|
128 |
+
author={Nils Reimers, Iryna Gurevych},
|
129 |
+
journal={https://arxiv.org/abs/1908.10084},
|
130 |
+
year={2019}
|
131 |
+
}
|
132 |
+
|
133 |
|
134 |
+
@article{martin2020camembert,
|
135 |
+
title={CamemBERT: a Tasty French Language Mode},
|
136 |
+
author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
|
137 |
+
journal={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
|
138 |
+
year={2020}
|
139 |
+
}
|