rufimelo commited on
Commit
3305a71
1 Parent(s): b4d0e73

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +103 -63
README.md CHANGED
@@ -1,42 +1,84 @@
1
  ---
2
- pipeline_tag: sentence-similarity
 
 
3
  tags:
4
  - sentence-transformers
5
- - feature-extraction
6
- - sentence-similarity
7
  - transformers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  ---
9
 
10
- # {MODEL_NAME}
 
 
 
 
11
 
 
 
 
12
  This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search.
 
13
 
14
- <!--- Describe your model here -->
15
 
16
- ## Usage (Sentence-Transformers)
 
17
 
18
- Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
19
 
 
 
 
 
 
20
  ```
21
  pip install -U sentence-transformers
22
  ```
23
-
24
  Then you can use the model like this:
25
-
26
  ```python
27
  from sentence_transformers import SentenceTransformer
28
- sentences = ["This is an example sentence", "Each sentence is converted"]
29
 
30
- model = SentenceTransformer('{MODEL_NAME}')
31
  embeddings = model.encode(sentences)
32
  print(embeddings)
33
  ```
34
-
35
-
36
-
37
  ## Usage (HuggingFace Transformers)
38
- Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
39
-
40
  ```python
41
  from transformers import AutoTokenizer, AutoModel
42
  import torch
@@ -48,13 +90,12 @@ def mean_pooling(model_output, attention_mask):
48
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
49
  return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
50
 
51
-
52
  # Sentences we want sentence embeddings for
53
  sentences = ['This is an example sentence', 'Each sentence is converted']
54
 
55
  # Load model from HuggingFace Hub
56
- tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
57
- model = AutoModel.from_pretrained('{MODEL_NAME}')
58
 
59
  # Tokenize sentences
60
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
@@ -62,64 +103,63 @@ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tenso
62
  # Compute token embeddings
63
  with torch.no_grad():
64
  model_output = model(**encoded_input)
65
-
66
  # Perform pooling. In this case, mean pooling.
67
  sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
68
-
69
  print("Sentence embeddings:")
70
  print(sentence_embeddings)
71
  ```
72
 
73
 
74
-
75
- ## Evaluation Results
76
-
77
- <!--- Describe how your model was evaluated -->
78
-
79
- For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
80
-
81
-
82
- ## Training
83
- The model was trained with the parameters:
84
-
85
- **DataLoader**:
86
-
87
- `torch.utils.data.dataloader.DataLoader` of length 2226 with parameters:
88
  ```
89
- {'batch_size': 8, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
 
 
 
90
  ```
 
91
 
92
- **Loss**:
 
93
 
94
- `sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss`
95
 
96
- Parameters of the fit()-Method:
97
- ```
98
- {
99
- "epochs": 5,
100
- "evaluation_steps": 0,
101
- "evaluator": "NoneType",
102
- "max_grad_norm": 1,
103
- "optimizer_class": "<class 'transformers.optimization.AdamW'>",
104
- "optimizer_params": {
105
- "lr": 1e-05
106
- },
107
- "scheduler": "WarmupLinear",
108
- "steps_per_epoch": null,
109
- "warmup_steps": 1113,
110
- "weight_decay": 0.01
111
  }
112
- ```
113
 
 
 
 
 
 
 
 
 
114
 
115
- ## Full Model Architecture
116
- ```
117
- SentenceTransformer(
118
- (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
119
- (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
120
- )
121
- ```
122
 
123
- ## Citing & Authors
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
- <!--- Describe where people can find more information -->
 
1
  ---
2
+ language:
3
+ - pt
4
+ thumbnail: Portuguese BERT for the Legal Domain
5
  tags:
6
  - sentence-transformers
 
 
7
  - transformers
8
+ - bert
9
+ - pytorch
10
+ - sentence-similarity
11
+ license: mit
12
+ pipeline_tag: sentence-similarity
13
+ datasets:
14
+ - stjiris/portuguese-legal-sentences-v0
15
+ - assin
16
+ - assin2
17
+ - stsb_multi_mt
18
+ widget:
19
+ - source_sentence: "O advogado apresentou as provas ao juíz."
20
+ sentences:
21
+ - "O juíz leu as provas."
22
+ - "O juíz leu o recurso."
23
+ - "O juíz atirou uma pedra."
24
+ model-index:
25
+ - name: BERTimbau
26
+ results:
27
+ - task:
28
+ name: STS
29
+ type: STS
30
+ metrics:
31
+ - name: Pearson Correlation - assin Dataset
32
+ type: Pearson Correlation
33
+ value: 0.7736839315650635
34
+ - name: Pearson Correlation - assin2 Dataset
35
+ type: Pearson Correlation
36
+ value: 0.8160303683026692
37
+ - name: Pearson Correlation - stsb_multi_mt pt Dataset
38
+ type: Pearson Correlation
39
+ value: 0.8549555966003046
40
+ - name: Pearson Correlation - IRIS sts Dataset
41
+ type: Pearson Correlation
42
+ value: 0.7873379128441461
43
  ---
44
 
45
+ [![INESC-ID](https://www.inesc-id.pt/wp-content/uploads/2019/06/INESC-ID-logo_01.png)](https://www.inesc-id.pt/projects/PR07005/)
46
+
47
+ [![A Semantic Search System for Supremo Tribunal de Justiça](https://rufimelo99.github.io/SemanticSearchSystemForSTJ/_static/logo.png)](https://rufimelo99.github.io/SemanticSearchSystemForSTJ/)
48
+
49
+ Work developed as part of [Project IRIS](https://www.inesc-id.pt/projects/PR07005/).
50
 
51
+ Thesis: [A Semantic Search System for Supremo Tribunal de Justiça](https://rufimelo99.github.io/SemanticSearchSystemForSTJ/)
52
+
53
+ # stjiris/bert-large-portuguese-cased-legal-tsdae-mkd-nli-sts-v1 (Legal BERTimbau)
54
  This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search.
55
+ stjiris/bert-large-portuguese-cased-legal-tsdae-mkd-nli-sts-v1 derives from stjiris/bert-large-portuguese-cased-legal-mlm (legal variant of [BERTimbau](https://huggingface.co/neuralmind/bert-large-portuguese-cased) large).
56
 
57
+ It was trained using the TSDAE technique with a learning rate 1e-5 [Legal Sentences from +-30000 documents](https://huggingface.co/datasets/stjiris/portuguese-legal-sentences-v1.0) 21.2k training steps (best performance for our semantic search system implementation)
58
 
59
+ This model was subjected to Multilingual Knowledge Distillation technique (mkd). For the Multilingual Knowledge Distillation process, the teacher model was 'sentence-transformers/stsb-roberta-large', the supposed supported language as English and the language to learn was portuguese
60
+ The dataset used was: TED 2020 – Parallel Sentences Corpus. TED 2020 contains around 4000 TED and TED-X transcripts from July 2020. These transcripts were translated by volunteers into more than 100 languages, adding up to a total of 10 544 174 sentences.
61
 
62
+ The model was presented to NLI data. 16 batch size, 2e-5 lr
63
 
64
+ It was trained for Semantic Textual Similarity, being submitted to a fine tuning stage with the [assin](https://huggingface.co/datasets/assin), [assin2](https://huggingface.co/datasets/assin2), [stsb_multi_mt pt](https://huggingface.co/datasets/stsb_multi_mt) datasets. 'lr': 1e-5
65
+
66
+
67
+ ## Usage (Sentence-Transformers)
68
+ Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
69
  ```
70
  pip install -U sentence-transformers
71
  ```
 
72
  Then you can use the model like this:
 
73
  ```python
74
  from sentence_transformers import SentenceTransformer
75
+ sentences = ["Isto é um exemplo", "Isto é um outro exemplo"]
76
 
77
+ model = SentenceTransformer('stjiris/bert-large-portuguese-cased-legal-tsdae-mkd-nli-sts-v1')
78
  embeddings = model.encode(sentences)
79
  print(embeddings)
80
  ```
 
 
 
81
  ## Usage (HuggingFace Transformers)
 
 
82
  ```python
83
  from transformers import AutoTokenizer, AutoModel
84
  import torch
 
90
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
91
  return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
92
 
 
93
  # Sentences we want sentence embeddings for
94
  sentences = ['This is an example sentence', 'Each sentence is converted']
95
 
96
  # Load model from HuggingFace Hub
97
+ tokenizer = AutoTokenizer.from_pretrained('stjiris/bert-large-portuguese-cased-legal-tsdae-mkd-nli-sts-v1')
98
+ model = AutoModel.from_pretrained('stjiris/bert-large-portuguese-cased-legal-tsdae-mkd-nli-sts-v1')
99
 
100
  # Tokenize sentences
101
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
 
103
  # Compute token embeddings
104
  with torch.no_grad():
105
  model_output = model(**encoded_input)
 
106
  # Perform pooling. In this case, mean pooling.
107
  sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
 
108
  print("Sentence embeddings:")
109
  print(sentence_embeddings)
110
  ```
111
 
112
 
113
+ ## Full Model Architecture
 
 
 
 
 
 
 
 
 
 
 
 
 
114
  ```
115
+ SentenceTransformer(
116
+ (0): Transformer({'max_seq_length': 514, 'do_lower_case': False}) with Transformer model: BertModel
117
+ (1): Pooling({'word_embedding_dimension': 1028, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False})
118
+ )
119
  ```
120
+ ## Citing & Authors
121
 
122
+ ### Contributions
123
+ [@rufimelo99](https://github.com/rufimelo99)
124
 
125
+ If you use this work, please cite:
126
 
127
+ ```bibtex
128
+ @inproceedings{MeloSemantic,
129
+ author = {Melo, Rui and Santos, Professor Pedro Alexandre and Dias, Professor Jo{\~ a}o},
130
+ title = {A {Semantic} {Search} {System} for {Supremo} {Tribunal} de {Justi}{\c c}a},
 
 
 
 
 
 
 
 
 
 
 
131
  }
 
132
 
133
+ @inproceedings{souza2020bertimbau,
134
+ author = {F{\'a}bio Souza and
135
+ Rodrigo Nogueira and
136
+ Roberto Lotufo},
137
+ title = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
138
+ booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
139
+ year = {2020}
140
+ }
141
 
142
+ @inproceedings{fonseca2016assin,
143
+ title={ASSIN: Avaliacao de similaridade semantica e inferencia textual},
144
+ author={Fonseca, E and Santos, L and Criscuolo, Marcelo and Aluisio, S},
145
+ booktitle={Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal},
146
+ pages={13--15},
147
+ year={2016}
148
+ }
149
 
150
+ @inproceedings{real2020assin,
151
+ title={The assin 2 shared task: a quick overview},
152
+ author={Real, Livy and Fonseca, Erick and Oliveira, Hugo Goncalo},
153
+ booktitle={International Conference on Computational Processing of the Portuguese Language},
154
+ pages={406--412},
155
+ year={2020},
156
+ organization={Springer}
157
+ }
158
+ @InProceedings{huggingface:dataset:stsb_multi_mt,
159
+ title = {Machine translated multilingual STS benchmark dataset.},
160
+ author={Philip May},
161
+ year={2021},
162
+ url={https://github.com/PhilipMay/stsb-multi-mt}
163
+ }
164
 
165
+ ```