patrickvonplaten
commited on
Commit
•
891192d
1
Parent(s):
bc5d071
Update README.md
Browse files
README.md
CHANGED
@@ -100,22 +100,21 @@ The following table summarizes the results for [fnet-base](https://huggingface.c
|
|
100 |
| WNLI | [00:02:37](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | [00:03:23](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli)|
|
101 |
| SUM | 16:30:45 | 24:23:56 |
|
102 |
|
103 |
-
|
104 |
-
| Task | MNLI-(m/mm) | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE | WNLI | SUM |
|
105 |
-
|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:----:|:-------:|
|
106 |
-
|FNet-base (PyTorch)| [06:40:55](https://huggingface.co/gchhablani/fnet-base-finetuned-mnli)| [06:21:16](https://huggingface.co/gchhablani/fnet-base-finetuned-qqp) | [01:48:22](https://huggingface.co/gchhablani/fnet-base-finetuned-qnli) | [01:09:27](https://huggingface.co/gchhablani/fnet-base-finetuned-sst2) | [00:09:47](https://huggingface.co/gchhablani/fnet-base-finetuned-cola) | [00:07:09](https://huggingface.co/gchhablani/fnet-base-finetuned-stsb) | [00:07:48](https://huggingface.co/gchhablani/fnet-base-finetuned-mrpc) | [00:03:24](https://huggingface.co/gchhablani/fnet-base-finetuned-rte) | [00:02:37](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | 16:30:45 |
|
107 |
-
|Bert-base (PyTorch)| [09:52:33](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mnli)| [09:25:01](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qqp) | [02:40:22](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qnli) | [01:42:17](https://huggingface.co/gchhablani/bert-base-cased-finetuned-sst2) | [00:14:20](https://huggingface.co/gchhablani/bert-base-cased-finetuned-cola) | [00:10:24](https://huggingface.co/gchhablani/bert-base-cased-finetuned-stsb) | [00:11:12](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mrpc) | [00:04:51](https://huggingface.co/gchhablani/bert-base-cased-finetuned-rte) | [00:03:23](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli) | 24:23:56 |
|
108 |
-
|
109 |
-
On average the PyTorch version of FNet-base requires *ca.* 30% less time for GLUE fine-tuning on GPU.
|
110 |
|
111 |
The following table summarizes the results for [fnet-base](https://huggingface.co/google/fnet-base) (called *FNet (PyTorch) - Reproduced*) and [bert-base-cased](https://hf.co/models/bert-base-cased) (called *Bert (PyTorch) - Reproduced*) in terms of performance and compares it to the reported performance of the official FNet-base model (called *FNet (Flax) - Official*).
|
112 |
|
113 |
-
| Task
|
114 |
-
|
115 |
-
|
|
116 |
-
|
|
117 |
-
|
|
118 |
-
|
|
|
|
|
|
|
|
|
|
|
|
119 |
|
120 |
We can see that FNet-base achieves around 93% of BERT-base's performance on average.
|
121 |
|
|
|
100 |
| WNLI | [00:02:37](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | [00:03:23](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli)|
|
101 |
| SUM | 16:30:45 | 24:23:56 |
|
102 |
|
103 |
+
On average the PyTorch version of FNet-base requires *ca.* 32% less time for GLUE fine-tuning on GPU.
|
|
|
|
|
|
|
|
|
|
|
|
|
104 |
|
105 |
The following table summarizes the results for [fnet-base](https://huggingface.co/google/fnet-base) (called *FNet (PyTorch) - Reproduced*) and [bert-base-cased](https://hf.co/models/bert-base-cased) (called *Bert (PyTorch) - Reproduced*) in terms of performance and compares it to the reported performance of the official FNet-base model (called *FNet (Flax) - Official*).
|
106 |
|
107 |
+
| Task/Model | Metric | FNet-base (PyTorch) | Bert-base (PyTorch) | FNet-Base (Flax - official) |
|
108 |
+
| MNLI-(m/mm) | Accuracy or Match/Mismatch | [76.75](https://huggingface.co/gchhablani/fnet-base-finetuned-mnli) | [84.10](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mnli) | 72/73 |
|
109 |
+
| QQP | mean(Accuracy,F1) | [86.5](https://huggingface.co/gchhablani/fnet-base-finetuned-qqp) | [89.26](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qqp) | 83 |
|
110 |
+
| QNLI | Accuracy | [84.39](https://huggingface.co/gchhablani/fnet-base-finetuned-qnli) | [90.99](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qnli) | 80 |
|
111 |
+
| SST-2 | Accuracy | [89.45](https://huggingface.co/gchhablani/fnet-base-finetuned-sst2) | [92.32](https://huggingface.co/gchhablani/bert-base-cased-finetuned-sst2) | 95 |
|
112 |
+
| CoLA | Matthews corr or Accuracy | [35.94](https://huggingface.co/gchhablani/fnet-base-finetuned-cola) | [59.57](https://huggingface.co/gchhablani/bert-base-cased-finetuned-cola) | 69 |
|
113 |
+
| STS-B | Spearman corr. | [82.19](https://huggingface.co/gchhablani/fnet-base-finetuned-stsb) | [88.98](https://huggingface.co/gchhablani/bert-base-cased-finetuned-stsb) | 79 |
|
114 |
+
| MRPC | mean(F1/Accuracy) | [81.15](https://huggingface.co/gchhablani/fnet-base-finetuned-mrpc) | [88.15](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mrpc) | 76 |
|
115 |
+
| RTE | Accuracy | [62.82](https://huggingface.co/gchhablani/fnet-base-finetuned-rte) | [67.15](https://huggingface.co/gchhablani/bert-base-cased-finetuned-rte) | 63 |
|
116 |
+
| WNLI | Accuracy | [54.93](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | [46.48](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli) | - |
|
117 |
+
| Avg | - | - | - | 76.7 |
|
118 |
|
119 |
We can see that FNet-base achieves around 93% of BERT-base's performance on average.
|
120 |
|