raynardj commited on
Commit
1c50d4f
1 Parent(s): 6812990

1st README for this model

Browse files
Files changed (1) hide show
  1. README.md +98 -0
README.md ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - ner
6
+ - gene
7
+ - protein
8
+ - rna
9
+ - bioinfomatics
10
+ license: apache-2.0
11
+ datasets:
12
+ - jnlpba
13
+ widget:
14
+ - text: "It consists of 25 exons encoding a 1,278-amino acid glycoprotein that is composed of 13 transmembrane domains"
15
+ ---
16
+
17
+ # NER to find Gene & Gene products
18
+ > The model was trained on jnlpba dataset
19
+
20
+ All the labels, the possible token classes.
21
+ ```json
22
+ {"label2id": {
23
+ "DNA": 2,
24
+ "O": 0,
25
+ "RNA": 5,
26
+ "cell_line": 4,
27
+ "cell_type": 3,
28
+ "protein": 1
29
+ }
30
+ }
31
+ ```
32
+
33
+ Notice, we removed the 'B-','I-' etc from data label.🗡
34
+
35
+ ## This is the template we suggest for using the model
36
+ ```python
37
+ from transformers import pipeline
38
+
39
+ ner = pipeline(task="ner",model="raynardj/ner-gene-gp", tokenizer="raynardj/ner-gene-gp")
40
+ ner("Your text", aggregation_strategy="first")
41
+ ```
42
+ And here is to make your output more consecutive ⭐️
43
+
44
+ ```python
45
+ import pandas as pd
46
+ from transformers import AutoTokenizer
47
+ tokenizer = AutoTokenizer.from_pretrained("raynardj/ner-gene-gp")
48
+
49
+ def clean_output(outputs):
50
+ results = []
51
+ current = []
52
+ last_idx = 0
53
+ # make to sub group by position
54
+ for output in outputs:
55
+ if output["index"]-1==last_idx:
56
+ current.append(output)
57
+ else:
58
+ results.append(current)
59
+ current = [output, ]
60
+ last_idx = output["index"]
61
+ if len(current)>0:
62
+ results.append(current)
63
+
64
+ # from tokens to string
65
+ strings = []
66
+ for c in results:
67
+ tokens = []
68
+ starts = []
69
+ ends = []
70
+ for o in c:
71
+ tokens.append(o['word'])
72
+ starts.append(o['start'])
73
+ ends.append(o['end'])
74
+
75
+ new_str = tokenizer.convert_tokens_to_string(tokens)
76
+ if new_str!='':
77
+ strings.append(dict(
78
+ word=new_str,
79
+ start = min(starts),
80
+ end = max(ends),
81
+ entity = c[0]['entity']
82
+ ))
83
+ return strings
84
+
85
+ def entity_table(pipeline, **pipeline_kw):
86
+ if "aggregation_strategy" not in pipeline_kw:
87
+ pipeline_kw["aggregation_strategy"] = "first"
88
+ def create_table(text):
89
+ return pd.DataFrame(
90
+ clean_output(
91
+ pipeline(text, **pipeline_kw)
92
+ )
93
+ )
94
+ return create_table
95
+
96
+ # will return a dataframe
97
+ entity_table(ner)(YOUR_VERY_CONTENTFUL_TEXT)
98
+ ```