---
license: apache-2.0
base_model: google/flan-t5-base
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: flan-t5-base-spelling
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# flan-t5-base-spelling

This model is a fine-tuned version of [google/flan-t5-base](https://huggingface.co/google/flan-t5-base)
on the wiki.en dataset from [oliverguhr/spelling](https://github.com/oliverguhr/spelling).

## Model description

This an experimental model that should be capable of fixing typos and punctuation.

```python
pipe = pipeline(
    "text2text-generation",
    model="jbochi/flan-t5-base-spelling",
    token=access_token
)
   
def fix_spelling(input_sentence):
    output = pipe("proofread: " + input_sentence, max_length=100)
    return output[0]['generated_text']
   
print(fix_spelling("lets do a comparsion"))
#lets do a comparison
```

## Intended uses & limitations

Intented for research purposes.

- It may produce artifacts.
- It doesn't support languages other than English.
- It was fine-tuned with a `max_length` of 100 tokens.

## Training and evaluation data

Data from [oliverguhr/spelling](https://github.com/oliverguhr/spelling), with a "proofread: " prefix added to every example.

The model was only evaluated on the first 100 test examples only during training.

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001 (probably too high)
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:------:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| 0.412         | 0.01  | 1000   | 0.3298          | 95.7852 | 91.7559 | 95.7602 | 95.7515   | 33.6    |
| 0.3593        | 0.01  | 2000   | 0.3301          | 95.8348 | 91.8272 | 95.8136 | 95.8144   | 33.55   |
| 0.3558        | 0.02  | 3000   | 0.3235          | 95.7455 | 91.6348 | 95.7048 | 95.7262   | 33.51   |
| 0.3242        | 0.03  | 4000   | 0.3359          | 95.863  | 91.8177 | 95.8296 | 95.8436   | 33.49   |
| 0.313         | 0.03  | 5000   | 0.3359          | 95.6728 | 91.4223 | 95.6613 | 95.6584   | 33.54   |
| 0.3123        | 0.04  | 6000   | 0.3441          | 95.8138 | 91.8344 | 95.7997 | 95.8025   | 33.6    |
| 0.3099        | 0.05  | 7000   | 0.3321          | 95.8138 | 91.8344 | 95.7997 | 95.8025   | 33.62   |
| 0.2958        | 0.05  | 8000   | 0.3269          | 95.8108 | 91.664  | 95.7949 | 95.7904   | 33.54   |
| 0.2749        | 0.06  | 9000   | 0.3237          | 95.758  | 91.7913 | 95.7622 | 95.7659   | 33.59   |
| 0.2841        | 0.07  | 10000  | 0.3114          | 95.8508 | 91.8853 | 95.8329 | 95.832    | 33.51   |
| 0.2772        | 0.07  | 11000  | 0.3206          | 95.8584 | 91.9881 | 95.853  | 95.864    | 33.54   |
| 0.2875        | 0.08  | 12000  | 0.3164          | 95.824  | 91.8821 | 95.8201 | 95.8307   | 33.56   |
| 0.3008        | 0.09  | 13000  | 0.3202          | 95.9349 | 92.075  | 95.9245 | 95.9207   | 33.55   |
| 0.288         | 0.09  | 14000  | 0.3140          | 95.7841 | 91.7283 | 95.7548 | 95.7585   | 33.42   |
| 0.2866        | 0.1   | 15000  | 0.3207          | 95.8259 | 91.8816 | 95.8057 | 95.8181   | 33.57   |
| 0.284         | 0.11  | 16000  | 0.3177          | 95.8209 | 91.8465 | 95.7971 | 95.7896   | 33.59   |
| 0.2574        | 0.11  | 17000  | 0.3146          | 95.8119 | 91.8995 | 95.7928 | 95.794    | 33.55   |
| 0.2807        | 0.12  | 18000  | 0.3214          | 95.7925 | 91.8605 | 95.769  | 95.7734   | 33.55   |
| 0.2742        | 0.13  | 19000  | 0.3185          | 95.8752 | 91.9684 | 95.8473 | 95.8513   | 33.49   |
| 0.2784        | 0.13  | 20000  | 0.3237          | 95.8729 | 92.0086 | 95.8636 | 95.8659   | 33.53   |
| 0.2768        | 0.14  | 21000  | 0.3187          | 95.6921 | 91.5779 | 95.6763 | 95.6681   | 33.44   |
| 0.2789        | 0.15  | 22000  | 0.3245          | 95.786  | 91.8861 | 95.7659 | 95.7607   | 33.5    |
| 0.2422        | 0.15  | 23000  | 0.3285          | 95.8421 | 91.9532 | 95.8388 | 95.8337   | 33.59   |
| 0.2838        | 0.16  | 24000  | 0.3186          | 95.5789 | 91.3976 | 95.5557 | 95.5694   | 33.56   |
| 0.2603        | 0.17  | 25000  | 0.3268          | 95.7276 | 91.6634 | 95.7156 | 95.7154   | 33.55   |
| 0.2622        | 0.17  | 26000  | 0.3230          | 95.808  | 91.9242 | 95.8018 | 95.7992   | 33.58   |
| 0.264         | 0.18  | 27000  | 0.3143          | 95.7982 | 91.8439 | 95.803  | 95.7941   | 33.6    |
| 0.26          | 0.19  | 28000  | 0.3245          | 95.7435 | 91.7233 | 95.7274 | 95.7203   | 33.6    |
| 0.2644        | 0.19  | 29000  | 0.3173          | 95.7982 | 91.8439 | 95.803  | 95.7941   | 33.56   |
| 0.2619        | 0.2   | 30000  | 0.3234          | 95.6744 | 91.5742 | 95.669  | 95.6593   | 33.57   |
| 0.2621        | 0.21  | 31000  | 0.3211          | 95.8658 | 91.9664 | 95.8593 | 95.8504   | 33.56   |
| 0.247         | 0.21  | 32000  | 0.3232          | 95.7248 | 91.4886 | 95.7127 | 95.7006   | 33.57   |
| 0.2428        | 0.22  | 33000  | 0.3206          | 95.8412 | 91.9314 | 95.8346 | 95.826    | 33.56   |
| 0.2389        | 0.23  | 34000  | 0.3125          | 95.7443 | 91.724  | 95.7435 | 95.7439   | 33.57   |
| 0.2634        | 0.23  | 35000  | 0.3205          | 95.8085 | 91.863  | 95.8091 | 95.805    | 33.56   |
| 0.2552        | 0.24  | 36000  | 0.3112          | 95.7519 | 91.7062 | 95.7286 | 95.744    | 33.54   |
| 0.2554        | 0.25  | 37000  | 0.3141          | 95.736  | 91.7453 | 95.7348 | 95.7258   | 33.56   |
| 0.2587        | 0.25  | 38000  | 0.3140          | 95.7572 | 91.6578 | 95.7428 | 95.7436   | 33.48   |
| 0.2521        | 0.26  | 39000  | 0.3146          | 95.7416 | 91.4858 | 95.7294 | 95.7293   | 33.47   |
| 0.2625        | 0.27  | 40000  | 0.3175          | 95.69   | 91.5155 | 95.6853 | 95.6856   | 33.53   |
| 0.2459        | 0.27  | 41000  | 0.3094          | 95.7464 | 91.7371 | 95.7353 | 95.7386   | 33.57   |
| 0.245         | 0.28  | 42000  | 0.3132          | 95.7602 | 91.7861 | 95.7498 | 95.7554   | 33.61   |
| 0.2403        | 0.29  | 43000  | 0.3169          | 95.6634 | 91.593  | 95.6744 | 95.6835   | 33.57   |
| 0.2516        | 0.29  | 44000  | 0.3146          | 95.6702 | 91.4878 | 95.6453 | 95.6495   | 33.53   |
| 0.2463        | 0.3   | 45000  | 0.3082          | 95.6617 | 91.5631 | 95.6687 | 95.6628   | 33.61   |
| 0.23          | 0.31  | 46000  | 0.3109          | 95.7859 | 91.8242 | 95.7675 | 95.7743   | 33.55   |
| 0.2486        | 0.31  | 47000  | 0.3116          | 95.8384 | 91.8854 | 95.8349 | 95.8324   | 33.5    |
| 0.249         | 0.32  | 48000  | 0.3129          | 95.7614 | 91.7998 | 95.7548 | 95.7663   | 33.53   |
| 0.2285        | 0.33  | 49000  | 0.3149          | 95.7684 | 91.7453 | 95.7513 | 95.761    | 33.56   |
| 0.2447        | 0.33  | 50000  | 0.3133          | 95.7226 | 91.7332 | 95.7089 | 95.7118   | 33.55   |
| 0.2374        | 0.34  | 51000  | 0.3096          | 95.7373 | 91.772  | 95.7205 | 95.7261   | 33.57   |
| 0.2361        | 0.35  | 52000  | 0.3156          | 95.8283 | 92.0269 | 95.8162 | 95.8259   | 33.6    |
| 0.2408        | 0.35  | 53000  | 0.3098          | 95.6854 | 91.7511 | 95.6702 | 95.6927   | 33.63   |
| 0.2419        | 0.36  | 54000  | 0.3140          | 95.5872 | 91.3338 | 95.5907 | 95.6066   | 33.54   |
| 0.2436        | 0.37  | 55000  | 0.3134          | 95.7498 | 91.8573 | 95.7465 | 95.7411   | 33.54   |
| 0.2396        | 0.37  | 56000  | 0.3138          | 95.7169 | 91.7169 | 95.698  | 95.7106   | 33.51   |
| 0.2315        | 0.38  | 57000  | 0.3122          | 95.809  | 91.9188 | 95.8006 | 95.7915   | 33.49   |
| 0.2298        | 0.39  | 58000  | 0.3181          | 95.6967 | 91.6519 | 95.6813 | 95.6906   | 33.49   |
| 0.2345        | 0.39  | 59000  | 0.3173          | 95.7213 | 91.6964 | 95.7225 | 95.7077   | 33.5    |
| 0.2323        | 0.4   | 60000  | 0.3169          | 95.6666 | 91.543  | 95.6482 | 95.6599   | 33.58   |
| 0.236         | 0.41  | 61000  | 0.3164          | 95.7845 | 91.8149 | 95.7699 | 95.7677   | 33.56   |
| 0.2246        | 0.41  | 62000  | 0.3110          | 95.6412 | 91.4598 | 95.6383 | 95.6356   | 33.5    |
| 0.2267        | 0.42  | 63000  | 0.3088          | 95.7137 | 91.6683 | 95.706  | 95.7034   | 33.53   |
| 0.232         | 0.43  | 64000  | 0.3105          | 95.7599 | 91.7777 | 95.7602 | 95.7566   | 33.56   |
| 0.2123        | 0.43  | 65000  | 0.3082          | 95.6892 | 91.6144 | 95.6855 | 95.6935   | 33.57   |
| 0.2195        | 0.44  | 66000  | 0.3053          | 95.7095 | 91.7089 | 95.7063 | 95.7029   | 33.54   |
| 0.2434        | 0.45  | 67000  | 0.3093          | 95.8082 | 91.8858 | 95.7912 | 95.7946   | 33.52   |
| 0.2336        | 0.45  | 68000  | 0.3050          | 95.814  | 91.8745 | 95.8045 | 95.7973   | 33.55   |
| 0.2326        | 0.46  | 69000  | 0.3029          | 95.7247 | 91.7338 | 95.7163 | 95.7136   | 33.51   |
| 0.2454        | 0.47  | 70000  | 0.3123          | 95.7778 | 91.7202 | 95.7531 | 95.7492   | 33.48   |
| 0.2402        | 0.47  | 71000  | 0.3090          | 95.7694 | 91.6795 | 95.766  | 95.7524   | 33.47   |
| 0.2233        | 0.48  | 72000  | 0.3100          | 95.7594 | 91.7237 | 95.7389 | 95.7391   | 33.53   |
| 0.2199        | 0.49  | 73000  | 0.3135          | 95.7177 | 91.6686 | 95.7014 | 95.7123   | 33.47   |
| 0.2205        | 0.49  | 74000  | 0.3116          | 95.714  | 91.5665 | 95.7022 | 95.7061   | 33.51   |
| 0.2178        | 0.5   | 75000  | 0.3120          | 95.7485 | 91.6867 | 95.7277 | 95.7411   | 33.54   |
| 0.2226        | 0.51  | 76000  | 0.3130          | 95.7285 | 91.6919 | 95.7199 | 95.7192   | 33.5    |
| 0.2199        | 0.51  | 77000  | 0.3123          | 95.7969 | 91.8832 | 95.7934 | 95.7782   | 33.48   |
| 0.2177        | 0.52  | 78000  | 0.3090          | 95.7166 | 91.7218 | 95.7148 | 95.7098   | 33.55   |
| 0.216         | 0.53  | 79000  | 0.3024          | 95.6977 | 91.689  | 95.7016 | 95.6875   | 33.53   |
| 0.2252        | 0.53  | 80000  | 0.3057          | 95.6664 | 91.6233 | 95.6616 | 95.6674   | 33.54   |
| 0.2209        | 0.54  | 81000  | 0.3057          | 95.4622 | 91.2615 | 95.4438 | 95.4563   | 33.54   |
| 0.2134        | 0.55  | 82000  | 0.3107          | 95.6903 | 91.6428 | 95.686  | 95.6862   | 33.53   |
| 0.2174        | 0.55  | 83000  | 0.3078          | 95.7232 | 91.6865 | 95.7109 | 95.7141   | 33.58   |
| 0.2217        | 0.56  | 84000  | 0.3062          | 95.6664 | 91.6233 | 95.6616 | 95.6674   | 33.55   |
| 0.2186        | 0.57  | 85000  | 0.3096          | 95.5492 | 91.3676 | 95.5382 | 95.5372   | 33.54   |
| 0.2192        | 0.57  | 86000  | 0.3070          | 95.6729 | 91.4616 | 95.6675 | 95.6665   | 33.52   |
| 0.2315        | 0.58  | 87000  | 0.3034          | 95.5492 | 91.3676 | 95.5382 | 95.5372   | 33.54   |
| 0.2248        | 0.59  | 88000  | 0.3023          | 95.7411 | 91.6705 | 95.7342 | 95.7318   | 33.55   |
| 0.2193        | 0.59  | 89000  | 0.3061          | 95.7364 | 91.709  | 95.7285 | 95.7354   | 33.57   |
| 0.2212        | 0.6   | 90000  | 0.3061          | 95.6604 | 91.5168 | 95.6399 | 95.6356   | 33.57   |
| 0.2287        | 0.61  | 91000  | 0.3073          | 95.7703 | 91.7829 | 95.7669 | 95.7617   | 33.57   |
| 0.239         | 0.61  | 92000  | 0.3063          | 95.7232 | 91.6865 | 95.7109 | 95.7141   | 33.59   |
| 0.2113        | 0.62  | 93000  | 0.3123          | 95.6757 | 91.4957 | 95.6738 | 95.6683   | 33.56   |
| 0.2259        | 0.63  | 94000  | 0.3110          | 95.6757 | 91.4957 | 95.6738 | 95.6683   | 33.56   |
| 0.2178        | 0.63  | 95000  | 0.3142          | 95.6548 | 91.4577 | 95.6529 | 95.6478   | 33.57   |
| 0.2288        | 0.64  | 96000  | 0.3051          | 95.7232 | 91.6865 | 95.7109 | 95.7141   | 33.59   |
| 0.2051        | 0.65  | 97000  | 0.3073          | 95.761  | 91.7916 | 95.7624 | 95.7573   | 33.59   |
| 0.2227        | 0.65  | 98000  | 0.3091          | 95.6803 | 91.526  | 95.6722 | 95.6768   | 33.56   |
| 0.2353        | 0.66  | 99000  | 0.3087          | 95.662  | 91.4535 | 95.6541 | 95.6524   | 33.55   |
| 0.217         | 0.67  | 100000 | 0.3046          | 95.6757 | 91.4957 | 95.6738 | 95.6683   | 33.56   |
| 0.1989        | 0.67  | 101000 | 0.3062          | 95.6757 | 91.4957 | 95.6738 | 95.6683   | 33.56   |
| 0.217         | 0.68  | 102000 | 0.3071          | 95.6757 | 91.4957 | 95.6738 | 95.6683   | 33.56   |
| 0.22          | 0.69  | 103000 | 0.3048          | 95.6757 | 91.4957 | 95.6738 | 95.6683   | 33.56   |
| 0.2202        | 0.69  | 104000 | 0.3081          | 95.6757 | 91.4957 | 95.6738 | 95.6683   | 33.56   |
| 0.2121        | 0.7   | 105000 | 0.3088          | 95.6265 | 91.4405 | 95.6194 | 95.6178   | 33.54   |
| 0.2137        | 0.71  | 106000 | 0.3096          | 95.7694 | 91.6795 | 95.766  | 95.7524   | 33.49   |
| 0.2261        | 0.71  | 107000 | 0.3041          | 95.7209 | 91.6199 | 95.7148 | 95.7064   | 33.47   |
| 0.2105        | 0.72  | 108000 | 0.3042          | 95.7209 | 91.6199 | 95.7148 | 95.7064   | 33.47   |
| 0.1974        | 0.73  | 109000 | 0.3045          | 95.6593 | 91.5597 | 95.656  | 95.6542   | 33.53   |
| 0.198         | 0.73  | 110000 | 0.3054          | 95.7694 | 91.6795 | 95.766  | 95.7524   | 33.49   |
| 0.2217        | 0.74  | 111000 | 0.3049          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.225         | 0.75  | 112000 | 0.3021          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2222        | 0.75  | 113000 | 0.3045          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2078        | 0.76  | 114000 | 0.3041          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2194        | 0.77  | 115000 | 0.3027          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2155        | 0.77  | 116000 | 0.3037          | 95.6593 | 91.5597 | 95.656  | 95.6542   | 33.53   |
| 0.2201        | 0.78  | 117000 | 0.3007          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2061        | 0.79  | 118000 | 0.3017          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2091        | 0.79  | 119000 | 0.3021          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.1921        | 0.8   | 120000 | 0.3036          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2013        | 0.81  | 121000 | 0.3033          | 95.7102 | 91.6135 | 95.7119 | 95.7038   | 33.55   |
| 0.2105        | 0.81  | 122000 | 0.3003          | 95.6511 | 91.473  | 95.6445 | 95.6453   | 33.54   |
| 0.2044        | 0.82  | 123000 | 0.2997          | 95.6511 | 91.473  | 95.6445 | 95.6453   | 33.55   |
| 0.2128        | 0.83  | 124000 | 0.2997          | 95.6511 | 91.473  | 95.6445 | 95.6453   | 33.55   |
| 0.2113        | 0.83  | 125000 | 0.3016          | 95.6317 | 91.4265 | 95.6215 | 95.6277   | 33.55   |
| 0.2078        | 0.84  | 126000 | 0.3003          | 95.6317 | 91.4265 | 95.6215 | 95.6277   | 33.55   |
| 0.2117        | 0.85  | 127000 | 0.3016          | 95.6511 | 91.473  | 95.6445 | 95.6453   | 33.55   |
| 0.2112        | 0.85  | 128000 | 0.3012          | 95.5576 | 91.1912 | 95.5377 | 95.5468   | 33.52   |
| 0.2035        | 0.86  | 129000 | 0.3018          | 95.6286 | 91.3629 | 95.608  | 95.6137   | 33.49   |
| 0.2228        | 0.87  | 130000 | 0.2999          | 95.6511 | 91.473  | 95.6445 | 95.6453   | 33.55   |
| 0.2079        | 0.87  | 131000 | 0.2999          | 95.6511 | 91.473  | 95.6445 | 95.6453   | 33.55   |
| 0.2145        | 0.88  | 132000 | 0.3004          | 95.6409 | 91.4248 | 95.6233 | 95.6332   | 33.53   |
| 0.1987        | 0.89  | 133000 | 0.3028          | 95.6409 | 91.4248 | 95.6233 | 95.6332   | 33.53   |
| 0.2045        | 0.89  | 134000 | 0.3043          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.1922        | 0.9   | 135000 | 0.3014          | 95.6313 | 91.4632 | 95.6214 | 95.6226   | 33.52   |
| 0.1956        | 0.91  | 136000 | 0.3003          | 95.6313 | 91.4632 | 95.6214 | 95.6226   | 33.52   |
| 0.2132        | 0.91  | 137000 | 0.3001          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.1989        | 0.92  | 138000 | 0.2998          | 95.6409 | 91.4248 | 95.6233 | 95.6332   | 33.53   |
| 0.2179        | 0.93  | 139000 | 0.2997          | 95.6409 | 91.4248 | 95.6233 | 95.6332   | 33.53   |
| 0.1921        | 0.93  | 140000 | 0.2994          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.2031        | 0.94  | 141000 | 0.3003          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.1961        | 0.95  | 142000 | 0.3021          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.2166        | 0.95  | 143000 | 0.3023          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.2105        | 0.96  | 144000 | 0.3021          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.2244        | 0.97  | 145000 | 0.3019          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.1998        | 0.97  | 146000 | 0.3017          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.2001        | 0.98  | 147000 | 0.3016          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.2152        | 0.99  | 148000 | 0.3015          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.1987        | 0.99  | 149000 | 0.3014          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |
| 0.2068        | 1.0   | 150000 | 0.3014          | 95.6848 | 91.5839 | 95.679  | 95.6777   | 33.54   |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu121
- Datasets 2.16.0
- Tokenizers 0.15.0