rufimelo commited on
Commit
c2e2565
1 Parent(s): e7faadd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +104 -62
README.md CHANGED
@@ -1,42 +1,83 @@
1
  ---
2
- pipeline_tag: sentence-similarity
 
 
3
  tags:
4
  - sentence-transformers
5
- - feature-extraction
6
- - sentence-similarity
7
  - transformers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  ---
9
 
10
- # {MODEL_NAME}
11
 
 
 
 
 
 
 
 
 
12
  This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search.
 
13
 
14
- <!--- Describe your model here -->
15
 
16
- ## Usage (Sentence-Transformers)
17
 
18
- Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
19
 
 
 
 
 
20
  ```
21
  pip install -U sentence-transformers
22
  ```
23
-
24
  Then you can use the model like this:
25
-
26
  ```python
27
  from sentence_transformers import SentenceTransformer
28
- sentences = ["This is an example sentence", "Each sentence is converted"]
29
 
30
- model = SentenceTransformer('{MODEL_NAME}')
31
  embeddings = model.encode(sentences)
32
  print(embeddings)
33
  ```
34
-
35
-
36
-
37
  ## Usage (HuggingFace Transformers)
38
- Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
39
-
40
  ```python
41
  from transformers import AutoTokenizer, AutoModel
42
  import torch
@@ -48,13 +89,12 @@ def mean_pooling(model_output, attention_mask):
48
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
49
  return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
50
 
51
-
52
  # Sentences we want sentence embeddings for
53
  sentences = ['This is an example sentence', 'Each sentence is converted']
54
 
55
  # Load model from HuggingFace Hub
56
- tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
57
- model = AutoModel.from_pretrained('{MODEL_NAME}')
58
 
59
  # Tokenize sentences
60
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
@@ -62,64 +102,66 @@ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tenso
62
  # Compute token embeddings
63
  with torch.no_grad():
64
  model_output = model(**encoded_input)
65
-
66
  # Perform pooling. In this case, mean pooling.
67
  sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
68
-
69
  print("Sentence embeddings:")
70
  print(sentence_embeddings)
71
  ```
72
 
73
 
 
 
 
 
 
 
 
74
 
75
- ## Evaluation Results
76
-
77
- <!--- Describe how your model was evaluated -->
78
-
79
- For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
80
-
81
-
82
- ## Training
83
- The model was trained with the parameters:
84
 
85
- **DataLoader**:
86
 
87
- `torch.utils.data.dataloader.DataLoader` of length 2226 with parameters:
88
- ```
89
- {'batch_size': 8, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
90
- ```
91
 
92
- **Loss**:
 
93
 
94
- `sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss`
95
 
96
- Parameters of the fit()-Method:
97
- ```
98
- {
99
- "epochs": 5,
100
- "evaluation_steps": 0,
101
- "evaluator": "NoneType",
102
- "max_grad_norm": 1,
103
- "optimizer_class": "<class 'transformers.optimization.AdamW'>",
104
- "optimizer_params": {
105
- "lr": 1e-05
106
- },
107
- "scheduler": "WarmupLinear",
108
- "steps_per_epoch": null,
109
- "warmup_steps": 1113,
110
- "weight_decay": 0.01
111
  }
112
- ```
113
 
 
 
 
 
 
 
 
 
114
 
115
- ## Full Model Architecture
116
- ```
117
- SentenceTransformer(
118
- (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
119
- (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
120
- )
121
- ```
122
 
123
- ## Citing & Authors
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
- <!--- Describe where people can find more information -->
 
1
  ---
2
+ language:
3
+ - pt
4
+ thumbnail: Portuguese BERT for the Legal Domain
5
  tags:
6
  - sentence-transformers
 
 
7
  - transformers
8
+ - bert
9
+ - pytorch
10
+ - sentence-similarity
11
+ license: mit
12
+ pipeline_tag: sentence-similarity
13
+ datasets:
14
+ - stjiris/portuguese-legal-sentences-v0
15
+ - assin
16
+ - assin2
17
+ - stsb_multi_mt
18
+ - stjiris/IRIS_sts
19
+ widget:
20
+ - source_sentence: "O advogado apresentou as provas ao juíz."
21
+ sentences:
22
+ - "O juíz leu as provas."
23
+ - "O juíz leu o recurso."
24
+ - "O juíz atirou uma pedra."
25
+ model-index:
26
+ - name: BERTimbau
27
+ results:
28
+ - task:
29
+ name: STS
30
+ type: STS
31
+ metrics:
32
+ - name: Pearson Correlation - assin Dataset
33
+ type: Pearson Correlation
34
+ value: 0.7763420633772975
35
+ - name: Pearson Correlation - assin2 Dataset
36
+ type: Pearson Correlation
37
+ value: 0.8067374216274927
38
+ - name: Pearson Correlation - stsb_multi_mt pt Dataset
39
+ type: Pearson Correlation
40
+ value: 0.8388993109077857
41
+ - name: Pearson Correlation - IRIS STS Dataset
42
+ type: Pearson Correlation
43
+ value: 0.7931353381814285
44
  ---
45
 
 
46
 
47
+ ![INESC-ID](https://www.inesc-id.pt/wp-content/uploads/2019/06/INESC-ID-logo_01.png)
48
+ ![A Semantic Search System for Supremo Tribunal de Justiça](https://rufimelo99.github.io/SemanticSearchSystemForSTJ/_static/logo.png)
49
+
50
+ Work developed as part of [Project IRIS](https://www.inesc-id.pt/projects/PR07005/).
51
+
52
+ Thesis: [A Semantic Search System for Supremo Tribunal de Justiça](https://rufimelo99.github.io/SemanticSearchSystemForSTJ/)
53
+
54
+ # stjiris/bert-large-portuguese-cased-legal-tsdae-gpl-nli-sts-v1 (Legal BERTimbau)
55
  This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search.
56
+ stjiris/bert-large-portuguese-cased-legal-tsdae derives from [BERTimbau](https://huggingface.co/neuralmind/bert-large-portuguese-cased) large.
57
 
58
+ It was trained using the TSDAE technique with a learning rate 1e-5 [Legal Sentences from +-30000 documents](https://huggingface.co/datasets/stjiris/portuguese-legal-sentences-v1.0) 212k training steps (best performance for our semantic search system implementation)
59
 
60
+ It was presented to Generative Pseudo Labeling training.
61
 
62
+ The model was presented to NLI data. 16 batch size, 2e-5 lr
63
 
64
+ It was trained for Semantic Textual Similarity, being submitted to a fine tuning stage with the [assin](https://huggingface.co/datasets/assin), [assin2](https://huggingface.co/datasets/assin2), [stsb_multi_mt pt](https://huggingface.co/datasets/stsb_multi_mt) datasets. 'lr': 1e-5
65
+
66
+ ## Usage (Sentence-Transformers)
67
+ Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
68
  ```
69
  pip install -U sentence-transformers
70
  ```
 
71
  Then you can use the model like this:
 
72
  ```python
73
  from sentence_transformers import SentenceTransformer
74
+ sentences = ["Isto é um exemplo", "Isto é um outro exemplo"]
75
 
76
+ model = SentenceTransformer('stjiris/bert-large-portuguese-cased-legal-tsdae-gpl-nli-sts-v1')
77
  embeddings = model.encode(sentences)
78
  print(embeddings)
79
  ```
 
 
 
80
  ## Usage (HuggingFace Transformers)
 
 
81
  ```python
82
  from transformers import AutoTokenizer, AutoModel
83
  import torch
 
89
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
90
  return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
91
 
 
92
  # Sentences we want sentence embeddings for
93
  sentences = ['This is an example sentence', 'Each sentence is converted']
94
 
95
  # Load model from HuggingFace Hub
96
+ tokenizer = AutoTokenizer.from_pretrained('stjiris/bert-large-portuguese-cased-legal-tsdae-gpl-nli-sts-v1')
97
+ model = AutoModel.from_pretrained('stjiris/bert-large-portuguese-cased-legal-tsdae-gpl-nli-sts-v1')
98
 
99
  # Tokenize sentences
100
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
 
102
  # Compute token embeddings
103
  with torch.no_grad():
104
  model_output = model(**encoded_input)
 
105
  # Perform pooling. In this case, mean pooling.
106
  sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
 
107
  print("Sentence embeddings:")
108
  print(sentence_embeddings)
109
  ```
110
 
111
 
112
+ ## Full Model Architecture
113
+ ```
114
+ SentenceTransformer(
115
+ (0): Transformer({'max_seq_length': 514, 'do_lower_case': False}) with Transformer model: BertModel
116
+ (1): Pooling({'word_embedding_dimension': 1028, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False})
117
+ )
118
+ ```
119
 
 
 
 
 
 
 
 
 
 
120
 
 
121
 
122
+ ## Citing & Authors
 
 
 
123
 
124
+ ### Contributions
125
+ [@rufimelo99](https://github.com/rufimelo99)
126
 
127
+ If you use this work, please cite:
128
 
129
+ ```bibtex
130
+ @inproceedings{MeloSemantic,
131
+ author = {Melo, Rui and Santos, Professor Pedro Alexandre and Dias, Professor Jo{\~ a}o},
132
+ title = {A {Semantic} {Search} {System} for {Supremo} {Tribunal} de {Justi}{\c c}a},
 
 
 
 
 
 
 
 
 
 
 
133
  }
 
134
 
135
+ @inproceedings{souza2020bertimbau,
136
+ author = {F{\'a}bio Souza and
137
+ Rodrigo Nogueira and
138
+ Roberto Lotufo},
139
+ title = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
140
+ booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
141
+ year = {2020}
142
+ }
143
 
144
+ @inproceedings{fonseca2016assin,
145
+ title={ASSIN: Avaliacao de similaridade semantica e inferencia textual},
146
+ author={Fonseca, E and Santos, L and Criscuolo, Marcelo and Aluisio, S},
147
+ booktitle={Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal},
148
+ pages={13--15},
149
+ year={2016}
150
+ }
151
 
152
+ @inproceedings{real2020assin,
153
+ title={The assin 2 shared task: a quick overview},
154
+ author={Real, Livy and Fonseca, Erick and Oliveira, Hugo Goncalo},
155
+ booktitle={International Conference on Computational Processing of the Portuguese Language},
156
+ pages={406--412},
157
+ year={2020},
158
+ organization={Springer}
159
+ }
160
+ @InProceedings{huggingface:dataset:stsb_multi_mt,
161
+ title = {Machine translated multilingual STS benchmark dataset.},
162
+ author={Philip May},
163
+ year={2021},
164
+ url={https://github.com/PhilipMay/stsb-multi-mt}
165
+ }
166
 
167
+ ```