initial

README.md

@@ -0,0 +1,134 @@
+---
+tags:
+- flair
+- token-classification
+- sequence-tagger-model
+language: nl
+datasets:
+- conll2003
+widget:
+- text: "George Washington ging naar Washington."
+---
+
+# Dutch NER in Flair (default model)
+
+This is the standard 4-class NER model for Dutch that ships with [Flair](https://github.com/flairNLP/flair/).
+
+F1-Score: **92,58** (CoNLL-03)
+
+Predicts 4 tags:
+
+| **tag**                        | **meaning** |
+|---------------------------------|-----------|
+| PER         | person name | 
+| LOC         | location name | 
+| ORG         | organization name | 
+| MISC         | other name | 
+
+Based on Transformer embeddings and LSTM-CRF.
+
+---
+# Demo: How to use in Flair
+
+Requires: **[Flair](https://github.com/flairNLP/flair/)** (`pip install flair`)
+
+```python
+from flair.data import Sentence
+from flair.models import SequenceTagger
+
+# load tagger
+tagger = SequenceTagger.load("flair/ner-dutch")
+
+# make example sentence
+sentence = Sentence("George Washington ging naar Washington")
+
+# predict NER tags
+tagger.predict(sentence)
+
+# print sentence
+print(sentence)
+
+# print predicted NER spans
+print('The following NER tags are found:')
+# iterate over entities and print
+for entity in sentence.get_spans('ner'):
+    print(entity)
+
+```
+
+This yields the following output:
+```
+Span [1,2]: "George Washington"   [− Labels: PER (0.997)]
+Span [5]: "Washington"   [− Labels: LOC (0.9996)]
+```
+
+So, the entities "*George Washington*" (labeled as a **person**) and "*Washington*" (labeled as a **location**) are found in the sentence "*George Washington ging naar Washington*". 
+
+
+---
+
+### Training: Script to train this model
+
+The following Flair script was used to train this model: 
+
+```python
+from flair.data import Corpus
+from flair.datasets import CONLL_03_DUTCH
+from flair.embeddings import WordEmbeddings, StackedEmbeddings, FlairEmbeddings
+
+
+# 1. get the corpus
+corpus: Corpus = CONLL_03_DUTCH()
+
+# 2. what tag do we want to predict?
+tag_type = 'ner'
+
+# 3. make the tag dictionary from the corpus
+tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)
+
+# 4. initialize embeddings
+embeddings = TransformerWordEmbeddings('wietsedv/bert-base-dutch-cased')
+
+# 5. initialize sequence tagger
+tagger: SequenceTagger = SequenceTagger(hidden_size=256,
+                                        embeddings=embeddings,
+                                        tag_dictionary=tag_dictionary,
+                                        tag_type=tag_type)
+
+# 6. initialize trainer
+trainer: ModelTrainer = ModelTrainer(tagger, corpus)
+
+# 7. run training
+trainer.train('resources/taggers/ner-dutch',
+              train_with_dev=True,
+              max_epochs=150)
+```
+
+
+---
+
+### Cite
+
+Please cite the following paper when using this model.
+
+```
+@inproceedings{akbik-etal-2019-flair,
+    title = "{FLAIR}: An Easy-to-Use Framework for State-of-the-Art {NLP}",
+    author = "Akbik, Alan  and
+      Bergmann, Tanja  and
+      Blythe, Duncan  and
+      Rasul, Kashif  and
+      Schweter, Stefan  and
+      Vollgraf, Roland",
+    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics (Demonstrations)",
+    year = "2019",
+    url = "https://www.aclweb.org/anthology/N19-4010",
+    pages = "54--59",
+}
+```
+
+---
+
+### Issues?
+
+The Flair issue tracker is available [here](https://github.com/flairNLP/flair/issues/).

deploy_endpoint_dix.ipynb

@@ -0,0 +1,168 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "initial_id",
+   "metadata": {
+    "collapsed": true,
+    "ExecuteTime": {
+     "end_time": "2023-10-05T07:20:29.202015200Z",
+     "start_time": "2023-10-05T07:20:29.190080700Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "True\n",
+      "0\n",
+      "<torch.cuda.device object at 0x0000028DAB1DB580>\n",
+      "1\n",
+      "NVIDIA GeForce RTX 3090\n"
+     ]
+    }
+   ],
+   "source": [
+    "import torch\n",
+    "\n",
+    "print(torch.cuda.is_available())     # Returns a bool indicating if CUDA is currently available.\n",
+    "print(torch.cuda.current_device())   # Returns the index of a currently selected device.\n",
+    "print(torch.cuda.device(0))          # Context-manager that changes the selected device.\n",
+    "print(torch.cuda.device_count())     # Returns the number of GPUs available.\n",
+    "print(torch.cuda.get_device_name(0)) # Gets the name of a device."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2023-10-05 10:37:05,350 SequenceTagger predicts: Dictionary with 20 tags: <unk>, O, S-ORG, S-MISC, B-PER, E-PER, S-PER, S-LOC, B-MISC, E-MISC, B-ORG, E-ORG, I-ORG, I-PER, B-LOC, I-LOC, E-LOC, I-MISC, <START>, <STOP>\n",
+      "non_holiday_pred [{'entity_group': 'PER', 'word': 'George Washington', 'start': 0, 'end': 17, 'score': 0.9970293045043945}, {'entity_group': 'LOC', 'word': 'Washington', 'start': 28, 'end': 38, 'score': 0.9996309280395508}]\n"
+     ]
+    }
+   ],
+   "source": [
+    "from handler import EndpointHandler\n",
+    "\n",
+    "# init handler\n",
+    "my_handler = EndpointHandler(path=\".\")\n",
+    "\n",
+    "# prepare sample payload\n",
+    "non_holiday_payload = {\"inputs\": \"George Washington ging naar Washington\"}\n",
+    "\n",
+    "\n",
+    "# test the handler\n",
+    "non_holiday_pred=my_handler(non_holiday_payload)\n",
+    "\n",
+    "\n",
+    "# show results\n",
+    "print(\"non_holiday_pred\", non_holiday_pred)\n",
+    "\n"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "ExecuteTime": {
+     "end_time": "2023-10-05T07:37:05.789680900Z",
+     "start_time": "2023-10-05T07:37:03.091564500Z"
+    }
+   },
+   "id": "a12c4a4792afc707"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "outputs": [],
+   "source": [
+    "from typing import Any, Dict, List\n",
+    "import os\n",
+    "from flair.data import Sentence\n",
+    "from flair.models import SequenceTagger"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "ExecuteTime": {
+     "end_time": "2023-10-05T07:36:53.389033800Z",
+     "start_time": "2023-10-05T07:36:53.382053200Z"
+    }
+   },
+   "id": "f411919d7d047065"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2023-10-05 10:36:58,072 SequenceTagger predicts: Dictionary with 20 tags: <unk>, O, S-ORG, S-MISC, B-PER, E-PER, S-PER, S-LOC, B-MISC, E-MISC, B-ORG, E-ORG, I-ORG, I-PER, B-LOC, I-LOC, E-LOC, I-MISC, <START>, <STOP>\n"
+     ]
+    }
+   ],
+   "source": [
+    "tagger = SequenceTagger.load('pytorch_model.bin')"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "ExecuteTime": {
+     "end_time": "2023-10-05T07:36:59.846440300Z",
+     "start_time": "2023-10-05T07:36:54.140093700Z"
+    }
+   },
+   "id": "8f497b3807de2e1"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "outputs": [
+    {
+     "data": {
+      "text/plain": "'0.12.2'"
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import flair\n",
+    "flair.__version__"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "ExecuteTime": {
+     "end_time": "2023-10-05T07:36:37.788428800Z",
+     "start_time": "2023-10-05T07:36:37.754490Z"
+    }
+   },
+   "id": "df243c485fd370b"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "name": "torch",
+   "language": "python",
+   "display_name": "torch"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

handler.py

@@ -0,0 +1,46 @@
+from typing import Any, Dict, List
+import os
+from flair.data import Sentence
+from flair.models import SequenceTagger
+
+class EndpointHandler():
+    def __init__(
+        self,
+        path: str,
+    ):
+        self.tagger = SequenceTagger.load(os.path.join(path,"pytorch_model.bin"))
+
+    def __call__(self, data: Dict[str, Any]) -> List[Dict[str, Any]]:
+        """
+        Args:
+            inputs (:obj:`str`):
+                a string containing some text
+        Return:
+            A :obj:`list`:. The object returned should be like [{"entity_group": "XXX", "word": "some word", "start": 3, "end": 6, "score": 0.82}] containing :
+                - "entity_group": A string representing what the entity is.
+                - "word": A substring of the original string that was detected as an entity.
+                - "start": the offset within `input` leading to `answer`. context[start:stop] == word
+                - "end": the ending offset within `input` leading to `answer`. context[start:stop] === word
+                - "score": A score between 0 and 1 describing how confident the model is for this entity.
+        """
+        inputs = data.pop("inputs", data)
+        sentence: Sentence = Sentence(inputs)
+
+        # Also show scores for recognized NEs
+        self.tagger.predict(sentence, label_name="predicted")
+
+        entities = []
+        for span in sentence.get_spans("predicted"):
+            if len(span.tokens) == 0:
+                continue
+            current_entity = {
+                "entity_group": span.tag,
+                "word": span.text,
+                "start": span.tokens[0].start_position,
+                "end": span.tokens[-1].end_position,
+                "score": span.score,
+            }
+
+            entities.append(current_entity)
+
+        return entities

loss.tsv

@@ -0,0 +1,151 @@
+EPOCH	TIMESTAMP	BAD_EPOCHS	LEARNING_RATE	TRAIN_LOSS
+1	14:01:30	0	0.1000	2.779733479175812
+2	14:03:00	0	0.1000	1.3395338043188438
+3	14:04:30	0	0.1000	1.0816994036874201
+4	14:06:01	0	0.1000	0.9621152551255674
+5	14:07:31	0	0.1000	0.8333948618835874
+6	14:09:01	0	0.1000	0.7716772701495733
+7	14:10:31	0	0.1000	0.7374578891911059
+8	14:12:01	0	0.1000	0.6992316849211342
+9	14:13:31	0	0.1000	0.6452770539328583
+10	14:15:01	1	0.1000	0.6467076227960423
+11	14:16:31	0	0.1000	0.620160369358511
+12	14:18:01	0	0.1000	0.5951391830658301
+13	14:19:31	0	0.1000	0.5727337152020544
+14	14:21:01	0	0.1000	0.5662796683163724
+15	14:22:32	0	0.1000	0.5356569104979181
+16	14:24:01	0	0.1000	0.5295078084764318
+17	14:25:31	0	0.1000	0.5199492649644868
+18	14:27:01	0	0.1000	0.5174320767832618
+19	14:28:31	0	0.1000	0.4972612695816236
+20	14:30:01	0	0.1000	0.4858957131194253
+21	14:31:30	1	0.1000	0.4954718519734521
+22	14:33:00	0	0.1000	0.475096299760362
+23	14:34:29	0	0.1000	0.45335076690739035
+24	14:35:58	0	0.1000	0.45071878243460617
+25	14:37:28	0	0.1000	0.4470571254053686
+26	14:38:57	0	0.1000	0.4351186317511094
+27	14:40:26	1	0.1000	0.435719607044489
+28	14:41:55	0	0.1000	0.43325941297743054
+29	14:43:24	0	0.1000	0.4213554557444703
+30	14:44:54	0	0.1000	0.41895739234920243
+31	14:46:23	1	0.1000	0.424293908655134
+32	14:47:53	0	0.1000	0.41112876903806994
+33	14:49:22	0	0.1000	0.4081363928368968
+34	14:50:52	0	0.1000	0.3930869156988258
+35	14:52:21	1	0.1000	0.4032204799927198
+36	14:53:50	0	0.1000	0.3917345738818503
+37	14:55:19	1	0.1000	0.4003689407920226
+38	14:56:48	0	0.1000	0.385329091319671
+39	14:58:17	0	0.1000	0.38347142062380785
+40	14:59:46	0	0.1000	0.3804049695515607
+41	15:01:14	0	0.1000	0.3764326793770505
+42	15:02:43	0	0.1000	0.3761323136142176
+43	15:04:12	1	0.1000	0.3851716780764425
+44	15:05:41	2	0.1000	0.3771476393326735
+45	15:07:11	0	0.1000	0.3616421103222757
+46	15:08:40	1	0.1000	0.3686442003800319
+47	15:10:09	0	0.1000	0.35834919491894224
+48	15:11:38	1	0.1000	0.3613178371618956
+49	15:13:07	0	0.1000	0.3519833675561807
+50	15:14:36	1	0.1000	0.35567070319611804
+51	15:16:05	0	0.1000	0.34542505874847756
+52	15:17:34	1	0.1000	0.34995765023761327
+53	15:19:03	0	0.1000	0.3352116090110224
+54	15:20:32	0	0.1000	0.3264291577868991
+55	15:22:01	1	0.1000	0.3271228834222525
+56	15:23:30	2	0.1000	0.33473273961462524
+57	15:25:00	3	0.1000	0.3365086276052345
+58	15:26:26	4	0.1000	0.33411403429559156
+59	15:27:48	0	0.0500	0.30620485559487953
+60	15:29:11	0	0.0500	0.28616658293793346
+61	15:30:37	0	0.0500	0.2821873968674077
+62	15:32:02	0	0.0500	0.26961317198653506
+63	15:33:25	0	0.0500	0.2660407587249055
+64	15:34:48	0	0.0500	0.2553254587782754
+65	15:36:12	1	0.0500	0.25559193193912505
+66	15:37:36	0	0.0500	0.24891968863642114
+67	15:38:59	1	0.0500	0.2530737343761656
+68	15:40:22	0	0.0500	0.2413989709992694
+69	15:41:45	1	0.0500	0.24797574456176188
+70	15:43:08	2	0.0500	0.242591819476782
+71	15:44:30	0	0.0500	0.23441884385851713
+72	15:45:54	1	0.0500	0.23821192036072414
+73	15:47:16	0	0.0500	0.23189536440066802
+74	15:48:39	0	0.0500	0.23109626505109998
+75	15:50:02	0	0.0500	0.21557108603545233
+76	15:51:24	1	0.0500	0.21703473255675063
+77	15:52:47	2	0.0500	0.21980291849527603
+78	15:54:09	3	0.0500	0.21841833248225032
+79	15:55:32	0	0.0500	0.21489991823322752
+80	15:56:55	0	0.0500	0.20688325222740825
+81	15:58:17	1	0.0500	0.22208200507184378
+82	15:59:40	2	0.0500	0.21044851431989262
+83	16:01:03	0	0.0500	0.2044361765568073
+84	16:02:25	0	0.0500	0.20330230971941582
+85	16:03:48	1	0.0500	0.2057137251027629
+86	16:05:11	0	0.0500	0.1991372063373908
+87	16:06:34	1	0.0500	0.2024117647136888
+88	16:07:57	2	0.0500	0.20274029105392277
+89	16:09:20	0	0.0500	0.19314180322819285
+90	16:10:43	0	0.0500	0.18950988753483847
+91	16:12:06	1	0.0500	0.19186331494432737
+92	16:13:29	2	0.0500	0.19439757827462423
+93	16:14:52	0	0.0500	0.18818266501284053
+94	16:16:15	0	0.0500	0.17917947142552107
+95	16:17:38	1	0.0500	0.18487144104945355
+96	16:19:01	2	0.0500	0.18518399219227652
+97	16:20:25	3	0.0500	0.18472332790111884
+98	16:21:48	0	0.0500	0.1754296123706855
+99	16:23:11	1	0.0500	0.18395276073621125
+100	16:24:33	2	0.0500	0.18039520557364846
+101	16:25:57	3	0.0500	0.18199163737077997
+102	16:27:20	0	0.0500	0.17518249248337542
+103	16:28:42	0	0.0500	0.17339555085596875
+104	16:30:05	1	0.0500	0.17634159061643812
+105	16:31:29	2	0.0500	0.17359274017473317
+106	16:32:52	0	0.0500	0.17105355417817578
+107	16:34:15	0	0.0500	0.1707773297260969
+108	16:35:38	0	0.0500	0.1698240860277771
+109	16:37:01	0	0.0500	0.16659483982202333
+110	16:38:24	0	0.0500	0.16050158162784373
+111	16:39:47	1	0.0500	0.1668573046596641
+112	16:41:09	0	0.0500	0.1591071498572317
+113	16:42:32	1	0.0500	0.16081542165106177
+114	16:43:55	2	0.0500	0.1627739645445194
+115	16:45:18	3	0.0500	0.16070217188352193
+116	16:46:41	4	0.0500	0.1620385132284246
+117	16:48:04	0	0.0250	0.14860385387626468
+118	16:49:27	0	0.0250	0.13816803481079573
+119	16:50:49	0	0.0250	0.13735185179135037
+120	16:52:13	0	0.0250	0.13719079037252654
+121	16:53:35	0	0.0250	0.13567532561401016
+122	16:54:57	1	0.0250	0.1358587311501177
+123	16:56:20	0	0.0250	0.1323100252618265
+124	16:57:43	1	0.0250	0.1347226249356555
+125	16:59:05	0	0.0250	0.12610901894732418
+126	17:00:28	0	0.0250	0.12457475662231446
+127	17:01:51	0	0.0250	0.12143644379754352
+128	17:03:14	1	0.0250	0.12776582111150792
+129	17:04:37	2	0.0250	0.12849602556636192
+130	17:06:00	3	0.0250	0.1244494532290687
+131	17:07:22	4	0.0250	0.12194784156277648
+132	17:08:45	0	0.0125	0.12072199874708796
+133	17:10:07	0	0.0125	0.11447634765735039
+134	17:11:29	1	0.0125	0.11593401964403625
+135	17:12:52	2	0.0125	0.11731710933809543
+136	17:14:14	0	0.0125	0.11113526996137559
+137	17:15:37	1	0.0125	0.11634638169382372
+138	17:17:00	2	0.0125	0.11969016508286835
+139	17:18:23	0	0.0125	0.11103729783820036
+140	17:19:45	1	0.0125	0.11450310367294866
+141	17:21:08	2	0.0125	0.11406106056056471
+142	17:22:31	3	0.0125	0.1144172125010409
+143	17:23:54	0	0.0125	0.10564635207510402
+144	17:25:16	0	0.0125	0.1036626111684192
+145	17:26:39	1	0.0125	0.10585466057826311
+146	17:28:02	2	0.0125	0.10889074587159686
+147	17:29:24	3	0.0125	0.11062939536089125
+148	17:30:47	4	0.0125	0.10591043005896429
+149	17:32:10	1	0.0063	0.1120014336748192
+150	17:33:33	0	0.0063	0.1027476362310923

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1daa8c197b464896aa5d1aafb5abd332f821d3ab80ab83a4d6b0c4ae819e4a2
+size 501524558