Any plans to train a version of your zero-shot models on ModernBERT? I'm finding that ModernBERT is a huge boost in speed, and a slight drop in performance vs. DeBERTa when I tune it. Not sure if the performance drop is because your zero-shot models were such a strong foundation for transfer learning, or the strength of DeBERTa architecture on NLI.
Mike Burnham
mlburnham
AI & ML interests
None yet
Recent Activity
replied to
MoritzLaurer's
post
4 days ago
Quite excited by the ModernBERT release! 0.15/0.4B small, 2T modern pre-training data and tokenizer with code, 8k context window, great efficient model for embeddings & classification!
This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D
Congrats @answerdotai, @LightOnIO and collaborators like @tomaarsen !
Paper and models here 👇https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
reacted
to
MoritzLaurer's
post
with đź‘Ť
12 days ago
Quite excited by the ModernBERT release! 0.15/0.4B small, 2T modern pre-training data and tokenizer with code, 8k context window, great efficient model for embeddings & classification!
This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D
Congrats @answerdotai, @LightOnIO and collaborators like @tomaarsen !
Paper and models here 👇https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
updated
a model
24 days ago
mlburnham/Political_DEBATE_base_v1.0
Organizations
None yet
mlburnham's activity
replied to
MoritzLaurer's
post
4 days ago
reacted to
MoritzLaurer's
post with đź‘Ť
12 days ago
Post
2557
Quite excited by the ModernBERT release! 0.15/0.4B small, 2T modern pre-training data and tokenizer with code, 8k context window, great efficient model for embeddings & classification!
This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D
Congrats @answerdotai , @LightOnIO and collaborators like @tomaarsen !
Paper and models here 👇https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D
Congrats @answerdotai , @LightOnIO and collaborators like @tomaarsen !
Paper and models here 👇https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
-
- 3 replies
updated
2
models
24 days ago
upvoted
a
paper
4 months ago
Add some basic metadata for the dataset
#2 opened 4 months ago
by
davanstrien
updated
a
dataset
4 months ago
[bot] Conversion to Parquet
#1 opened 6 months ago
by
parquet-converter
updated
2
models
5 months ago
updated
4
datasets
5 months ago
updated
6
datasets
6 months ago
mlburnham/adl_events_entailment
Viewer
•
Updated
•
1.67k
•
30
mlburnham/scad_event_entailment
Viewer
•
Updated
•
3.66k
•
31
mlburnham/bill_summary_entailment
Viewer
•
Updated
•
18.3k
•
36
mlburnham/polarizing_rhetoric
Viewer
•
Updated
•
1.56k
•
35
mlburnham/targeted_hatespeech_entailment
Viewer
•
Updated
•
22k
•
33
mlburnham/dehumanizing_hatespeech_entailment
Viewer
•
Updated
•
6.07k
•
29