--- license: apache-2.0 library_name: span-marker tags: - span-marker - token-classification - ner - named-entity-recognition pipeline_tag: token-classification model-index: - name: SpanMarker w. bert-base-cased on CrossNER by Tom Aarsen results: - task: type: token-classification name: Named Entity Recognition dataset: type: P3ps/Cross_ner name: CrossNER split: test revision: 7cecbbb3d2eb8c75c8571c53e5a5270cfd0c5a9e metrics: - type: f1 value: 0.8785 name: F1 - type: precision value: 0.8825 name: Precision - type: recall value: 0.8746 name: Recall datasets: - P3ps/Cross_ner language: - en metrics: - f1 - recall - precision --- # SpanMarker for Named Entity Recognition This is a [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) model that can be used for Named Entity Recognition. In particular, this SpanMarker model uses [bert-base-cased](https://huggingface.co/bert-base-cased) as the underlying encoder. See [train.py](train.py) for the training script. Is your data not (always) capitalized correctly? Then consider using the uncased variant of this model instead for better performance: [tomaarsen/span-marker-bert-base-uncased-cross-ner](https://huggingface.co/tomaarsen/span-marker-bert-base-uncased-cross-ner). ## Labels & Metrics | **Label** | **Examples** | **Precision** | **Recall** | **F1** | |:-------------------|---|---------------:|-----------:|-------:| | **all** | - | 88.25 | 87.46 | 87.85 | | academicjournal | "New Journal of Physics", "EPL", "European Physical Journal B" | 84.04 | 96.34 | 89.77 | | album | "Tellin' Stories", "Generation Terrorists", "Country Airs" | 90.71 | 85.81 | 88.19 | | algorithm | "LDA", "PCA", "gradient descent" | 76.27 | 79.65 | 77.92 | | astronomicalobject | "Earth", "Sun", "Halley's comet" | 92.00 | 93.24 | 92.62 | | award | "Nobel Prize for Literature", "Acamedy Award for Best Actress", "Mandelbrot's awards" | 87.14 | 92.51 | 89.74 | | band | "Clash", "Parliament Funkadelic", "Sly and the Family Stone" | 83.44 | 86.62 | 85.00 | | book | "Nietzsche contra Wagner" , "Dionysian-Dithyrambs", "The Rebel" | 73.71 | 82.69 | 77.95 | | chemicalcompound | "hydrogen sulfide", "Starch", "Lactic acid" | 71.21 | 71.21 | 71.21 | | chemicalelement | "potassium", "Fluorine", "Chlorine" | 84.00 | 70.00 | 76.36 | | conference | "SIGGRAPH", "IJCAI", "IEEE Transactions on Speech and Audio Processing" | 80.00 | 68.57 | 73.85 | | country | "United Arab Emirates", "U.S.", "Canada" | 81.72 | 86.81 | 84.19 | | discipline | "physics", "meteorology", "geography" | 48.39 | 55.56 | 51.72 | | election | "2004 Canadian federal election", "2006 Canadian federal election", "1999 Scottish Parliament election" | 96.61 | 97.85 | 97.23 | | enzyme | "RNA polymerase", "Phosphoinositide 3-kinase", "Protein kinase C" | 77.27 | 91.89 | 83.95 | | event | "Cannes Film Festival", "2019 Special Olympics World Summer Games", "2017 Western Iraq campaign" | 75.00 | 66.30 | 70.38 | | field | "computational imaging", "electronics", "information theory" | 89.80 | 83.02 | 86.27 | | literarygenre | "novel", "satire", "short story" | 70.24 | 68.60 | 69.41 | | location | "China", "BOMBAY", "Serbia" | 95.21 | 93.72 | 94.46 | | magazine | "The Atlantic", "The American Spectator", "Astounding Science Fiction" | 81.48 | 78.57 | 80.00 | | metrics | "BLEU", "precision", "DCG" | 72.53 | 81.48 | 76.74 | | misc | "Serbian", "Belgian", "The Birth of a Nation" | 81.69 | 74.08 | 77.70 | | musicalartist | "Chuck Burgi", "John Miceli", "John O'Reilly" | 79.67 | 87.11 | 83.23 | | musicalinstrument | "koto", "bubens", "def" | 66.67 | 22.22 | 33.33 | | musicgenre | "Christian rock", "Punk rock", "romantic melodicism" | 86.49 | 90.57 | 88.48 | | organisation | "IRISH TIMES", "Comintern", "Wimbledon" | 91.37 | 90.85 | 91.11 | | person | "Gong Zhichao", "Liu Lufung", "Margret Crowley" | 94.15 | 92.31 | 93.22 | | poem | "Historia destructionis Troiae", "I Am Joaquin", "The Snow Man" | 83.33 | 68.63 | 75.27 | | politicalparty | "New Democratic Party", "Bloc Québécois", "Liberal Party of Canada" | 87.50 | 90.17 | 88.82 | | politician | "Susan Kadis", "Simon Strelchik", "Lloyd Helferty" | 86.16 | 88.93 | 87.52 | | product | "AlphaGo", "WordNet", "Facial recognition system" | 60.82 | 70.24 | 65.19 | | programlang | "R", "C++", "Java" | 92.00 | 71.88 | 80.70 | | protein | "DNA methyltransferase", "tau protein", "Amyloid beta" | 60.29 | 59.42 | 59.85 | | researcher | "Sirovich", "Kirby", "Matthew Turk" | 87.50 | 78.65 | 82.84 | | scientist | "Matjaž Perc", "Cotton", "Singer" | 82.04 | 88.48 | 85.14 | | song | "Right Where I'm Supposed to Be", "Easy", "Three Times a Lady" | 84.78 | 90.70 | 87.64 | | task | "robot control", "elevator scheduling", "telecommunications" | 76.19 | 74.42 | 75.29 | | theory | "Big Bang", "general theory of relativity", "Ptolemaic planetary theories" | 100.00 | 16.67 | 28.57 | | university | "University of Göttingen", "Duke", "Imperial Academy of Sciences" | 77.14 | 91.01 | 83.51 | | writer | "Thomas Mann", "George Bernard Shaw", "Thomas Hardy" | 76.29 | 82.84 | 79.43 | ## Usage To use this model for inference, first install the `span_marker` library: ```bash pip install span_marker ``` You can then run inference with this model like so: ```python from span_marker import SpanMarkerModel # Download from the 🤗 Hub model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-bert-base-cross-ner") # Run inference entities = model.predict("Amelia Earhart flew her single engine Lockheed Vega 5B across the Atlantic to Paris.") ``` See the [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) repository for documentation and additional information on this library. ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-05 - train_batch_size: 32 - eval_batch_size: 32 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_ratio: 0.1 - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy | |:-------------:|:-----:|:----:|:---------------:|:-----------------:|:--------------:|:----------:|:----------------:| | 0.0521 | 0.25 | 200 | 0.0375 | 0.7149 | 0.6033 | 0.6544 | 0.8926 | | 0.0225 | 0.5 | 400 | 0.0217 | 0.8001 | 0.7878 | 0.7939 | 0.9400 | | 0.0189 | 0.75 | 600 | 0.0168 | 0.8526 | 0.8288 | 0.8405 | 0.9534 | | 0.0157 | 1.01 | 800 | 0.0160 | 0.8481 | 0.8366 | 0.8423 | 0.9543 | | 0.0116 | 1.26 | 1000 | 0.0158 | 0.8570 | 0.8568 | 0.8569 | 0.9582 | | 0.0119 | 1.51 | 1200 | 0.0145 | 0.8752 | 0.8550 | 0.8650 | 0.9607 | | 0.0102 | 1.76 | 1400 | 0.0145 | 0.8766 | 0.8555 | 0.8659 | 0.9601 | | 0.01 | 2.01 | 1600 | 0.0139 | 0.8744 | 0.8718 | 0.8731 | 0.9629 | | 0.0072 | 2.26 | 1800 | 0.0144 | 0.8748 | 0.8684 | 0.8716 | 0.9625 | | 0.0066 | 2.51 | 2000 | 0.0140 | 0.8803 | 0.8738 | 0.8770 | 0.9645 | | 0.007 | 2.76 | 2200 | 0.0138 | 0.8831 | 0.8739 | 0.8785 | 0.9644 | ### Framework versions - SpanMarker 1.2.4 - Transformers 4.31.0 - Pytorch 2.0.1+cu118 - Datasets 2.14.3 - Tokenizers 0.13.2