Norod78 Ezi commited on
Commit
a71dae1
1 Parent(s): 3429083

Model Card (#1)

Browse files

- Model Card (425a0dd4ea29d4c1116b93d2f05de9d5355ee55a)


Co-authored-by: Ezi Ozoani <Ezi@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +106 -2
README.md CHANGED
@@ -14,9 +14,113 @@ license: mit
14
 
15
  # hebrew-bad_wiki-gpt_neo-tiny
16
 
17
- Hebrew nonsense generation model which produces really bad wiki-abstract text. This model was fined tuned upon [hebrew-gpt_neo-tiny](https://huggingface.co/Norod78/hebrew-gpt_neo-tiny) which was previously trained using [EleutherAI's gpt-neo](https://github.com/EleutherAI/gpt-neo). Fine-tuning on the wiki-absract text was done using [@minimaxir](https://twitter.com/minimaxir)'s [aitextgen](https://github.com/minimaxir/aitextgen).
 
 
 
 
 
 
 
18
 
19
- ## Datasets
 
20
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  [Hebrew Wikipedia Dump](https://dumps.wikimedia.org/hewiki/latest/) (hewiki abstract) from May 2020
22
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
 
15
  # hebrew-bad_wiki-gpt_neo-tiny
16
 
17
+ ## Table of Contents
18
+ - [Model Details](#model-details)
19
+ - [Uses](#uses)
20
+ - [Risks, Limitations and Biases](#risks-limitations-and-biases)
21
+ - [Training](#training)
22
+ - [Evaluation](#evaluation)
23
+ - [Environmental Impact](#environmental-impact)
24
+ - [How to Get Started With the Model](#how-to-get-started-with-the-model)
25
 
26
+ ## Model Details
27
+ **Model Description:**
28
 
29
+ The model developer notes that the model is
30
+ > Hebrew nonsense generation model which produces really bad wiki-abstract text.
31
+
32
+
33
+ - **Developed by:** [Doron Adler](https://github.com/Norod)
34
+ - **Model Type:** Text Generation
35
+ - **Language(s):** Hebrew
36
+ - **License:** MIT
37
+ - **Resources for more information:**
38
+ - [GitHub Repo](https://github.com/Norod/hebrew-gpt_neo)
39
+ - [HuggingFace Space](https://huggingface.co/spaces/Norod78/Hebrew-GPT-Neo-Small)
40
+
41
+
42
+ ## Uses
43
+
44
+ #### Direct Use
45
+
46
+ This model can be used for text generation.
47
+
48
+ #### Misuse and Out-of-scope Use
49
+
50
+
51
+ ## Risks, Limitations and Biases
52
+ **CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
53
+
54
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
55
+
56
+
57
+
58
+
59
+ ## Training
60
+
61
+
62
+ #### Training Data
63
  [Hebrew Wikipedia Dump](https://dumps.wikimedia.org/hewiki/latest/) (hewiki abstract) from May 2020
64
 
65
+
66
+
67
+
68
+ #### Training Procedure
69
+
70
+
71
+ This model was fined tuned upon [hebrew-gpt_neo-tiny](https://huggingface.co/Norod78/hebrew-gpt_neo-tiny) which was previously trained using [EleutherAI's gpt-neo](https://github.com/EleutherAI/gpt-neo).
72
+
73
+ Fine-tuning on the wiki-absract text was done using [@minimaxir](https://twitter.com/minimaxir)'s [aitextgen](https://github.com/minimaxir/aitextgen).
74
+
75
+
76
+
77
+ ## Evaluation
78
+
79
+
80
+ #### Configs
81
+
82
+ Model configs for the hebrew-gpt_neo-tiny is available on the [hebrew-gpt_neo model github](https://github.com/Norod/hebrew-gpt_neo/tree/main/hebrew-gpt_neo-tiny/configs)
83
+
84
+ * **Activation Function:** gelu
85
+ * **Number_Head:** 12
86
+ * **Number_Vocab:** 50257
87
+ * **Train batch size:** 250
88
+ * **Eval batch size:** 64
89
+ * **Predict batch size:** 1
90
+
91
+
92
+
93
+
94
+ ## Environmental Impact
95
+
96
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type based on the [associated paper](https://arxiv.org/pdf/2105.09680.pdf).
97
+
98
+
99
+ - **Hardware Type:** [More information needed]
100
+
101
+ - **Hours used:** Unknown
102
+
103
+ - **Cloud Provider:** GCP tpu-v8s
104
+
105
+ - **Compute Region:** europe-west4
106
+
107
+ - **Carbon Emitted:** [More information needed]
108
+
109
+
110
+ ## How to Get Started With the Model
111
+
112
+ A Google Colab Notebook is also available [here](https://colab.research.google.com/github/Norod/hebrew-gpt_neo/blob/main/hebrew-gpt_neo-tiny/Norod78_hebrew_gpt_neo_tiny_Colab.ipynb)
113
+
114
+
115
+ ​​
116
+ ```
117
+ from transformers import AutoTokenizer, AutoModelForCausalLM
118
+
119
+ tokenizer = AutoTokenizer.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
120
+
121
+ model = AutoModelForCausalLM.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
122
+
123
+
124
+ ```
125
+
126
+