noahsantacruz commited on
Commit
2b0d3fa
1 Parent(s): 23f21b7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +63 -0
README.md CHANGED
@@ -21,6 +21,69 @@ model-index:
21
  type: f_score
22
  value: 0.8295994569
23
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
  | Feature | Description |
25
  | --- | --- |
26
  | **Name** | `he_ref_ner` |
 
21
  type: f_score
22
  value: 0.8295994569
23
  ---
24
+
25
+ See below for technical details about the model.
26
+
27
+ # Description
28
+
29
+ This model is a named entity recognition model that was trained to run on text that discusses Torah topics (e.g. dvar torahs, Torah blogs, translations of classic Torah texts etc.).
30
+
31
+ It detects the following types of entities:
32
+
33
+ | Label | Description
34
+ |---|---|
35
+ | Citation | Citations to Torah texts. See notes below. |
36
+
37
+ ## Notes on citation matches
38
+
39
+ - Final parentheses is not included in the match. E.g. if the citation is `בראשית (א:א)` then the final parentheses will not be included. We found that the model would get confused if the final parentheses was part of the entity. It is fairly simple to add it back in via a deterministic check.
40
+ - Only the first word of a dibur hamatchil is included in the match. E.g. `תוספות ד״ה אמר רבי עקיבא` only until the word `אמר` will be tagged. We found the model had trouble determining the end of the dibur hamatchil.
41
+ - See Ref part model for a model that can break down citations into chunks so it is simpler to parse them.
42
+
43
+ ## Using with Sefaria-Project
44
+
45
+ The [Sefaria-Project](https://github.com/Sefaria/Sefaria-Project) repo can use this model to return objects linked to objects in the Sefaria database. Non-citation entities are linked to `Topic` objects and citation entities are linked to `Ref` objects.
46
+
47
+ ### Configuring Sefaria-Project to use this model
48
+
49
+ The assumption is that Sefaria-Project is set up on your environment following the instructions in our [README](https://github.com/Sefaria/Sefaria-Project/blob/master/README.mkd).
50
+
51
+ In `local_settings.py`, modify the following lines:
52
+
53
+ ```python
54
+ RAW_REF_MODEL_BY_LANG_FILEPATH = {
55
+ "he": "/path/to/he-ref-ner model"
56
+ }
57
+ ```
58
+
59
+ ### Running the model with Sefaria-Project
60
+
61
+ The following code shows an example of instantiating the `Linker` object which uses the ML models and running the `Linker` with input.
62
+
63
+ ```python
64
+ import django
65
+ django.setup()
66
+ from sefaria.model.text import library
67
+
68
+ text = "משה קבל תורה מסיני (אבות פרק א משנה א)"
69
+ linker = library.get_linker("he")
70
+ doc = linker.link(text)
71
+
72
+ print("Named entities")
73
+ for resolved_named_entity in doc.resolved_named_entities:
74
+ print("---")
75
+ print("Text:", resolved_named_entity.raw_entity.text)
76
+ print("Topic Slug:", resolved_named_entity.topic.slug)
77
+
78
+ print("Citations")
79
+ for resolved_ref in doc.resolved_refs:
80
+ print("---")
81
+ print("Text:", resolved_ref.raw_entity.text)
82
+ print("Ref:", resolved_ref.ref.normal())
83
+ ```
84
+
85
+ # Technical Details
86
+
87
  | Feature | Description |
88
  | --- | --- |
89
  | **Name** | `he_ref_ner` |