SetFit

This is a SetFit model that can be used for Text Classification. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 5000 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("rkoh/setfit-bert")
# Run inference
preds = model("(Repealed). Author: Michael E. Mason, CPA")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	tensor(1)	tensor(370.1842)	tensor(52538)

Label	Training Sample Count
Purpose - Regulatory Objective	0
Scope and Applicability	0
Authority and Legal Basis	0
Administrative Details	0
Non-Purpose	0

Training Hyperparameters

batch_size: (32, 32)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0002	1	0.1006	-
0.0016	10	0.0759	-
0.0032	20	0.0767	-
0.0048	30	0.0852	-
0.0064	40	0.0765	-
0.008	50	0.078	-
0.0096	60	0.0734	-
0.0112	70	0.0687	-
0.0128	80	0.0566	-
0.0144	90	0.065	-
0.016	100	0.0583	-
0.0176	110	0.0584	-
0.0192	120	0.0466	-
0.0208	130	0.0661	-
0.0224	140	0.0583	-
0.024	150	0.0494	-
0.0256	160	0.0451	-
0.0272	170	0.0443	-
0.0288	180	0.0409	-
0.0304	190	0.0513	-
0.032	200	0.0415	-
0.0336	210	0.0413	-
0.0352	220	0.0478	-
0.0368	230	0.0319	-
0.0384	240	0.0273	-
0.04	250	0.0418	-
0.0416	260	0.0415	-
0.0432	270	0.0454	-
0.0448	280	0.0333	-
0.0464	290	0.0341	-
0.048	300	0.0504	-
0.0496	310	0.0296	-
0.0512	320	0.0293	-
0.0528	330	0.0263	-
0.0544	340	0.0292	-
0.056	350	0.0394	-
0.0576	360	0.0246	-
0.0592	370	0.0419	-
0.0608	380	0.0251	-
0.0624	390	0.02	-
0.064	400	0.0397	-
0.0656	410	0.0151	-
0.0672	420	0.0312	-
0.0688	430	0.0336	-
0.0704	440	0.0194	-
0.072	450	0.0251	-
0.0736	460	0.0167	-
0.0752	470	0.0203	-
0.0768	480	0.0158	-
0.0784	490	0.0165	-
0.08	500	0.0181	-
0.0816	510	0.0153	-
0.0832	520	0.0301	-
0.0848	530	0.0243	-
0.0864	540	0.0271	-
0.088	550	0.0185	-
0.0896	560	0.0221	-
0.0912	570	0.0171	-
0.0928	580	0.0284	-
0.0944	590	0.0335	-
0.096	600	0.0163	-
0.0976	610	0.0199	-
0.0992	620	0.0212	-
0.1008	630	0.0253	-
0.1024	640	0.0173	-
0.104	650	0.0376	-
0.1056	660	0.0135	-
0.1072	670	0.0216	-
0.1088	680	0.0279	-
0.1104	690	0.0126	-
0.112	700	0.0144	-
0.1136	710	0.0149	-
0.1152	720	0.0186	-
0.1168	730	0.0084	-
0.1184	740	0.0231	-
0.12	750	0.0152	-
0.1216	760	0.0174	-
0.1232	770	0.0235	-
0.1248	780	0.0144	-
0.1264	790	0.0081	-
0.128	800	0.0209	-
0.1296	810	0.014	-
0.1312	820	0.0208	-
0.1328	830	0.0146	-
0.1344	840	0.0159	-
0.136	850	0.0119	-
0.1376	860	0.0251	-
0.1392	870	0.0153	-
0.1408	880	0.0077	-
0.1424	890	0.0136	-
0.144	900	0.0131	-
0.1456	910	0.0058	-
0.1472	920	0.0146	-
0.1488	930	0.0186	-
0.1504	940	0.014	-
0.152	950	0.0127	-
0.1536	960	0.0074	-
0.1552	970	0.0246	-
0.1568	980	0.0137	-
0.1584	990	0.0061	-
0.16	1000	0.0067	-
0.1616	1010	0.0125	-
0.1632	1020	0.01	-
0.1648	1030	0.0116	-
0.1664	1040	0.0098	-
0.168	1050	0.0116	-
0.1696	1060	0.0051	-
0.1712	1070	0.0014	-
0.1728	1080	0.0056	-
0.1744	1090	0.0009	-
0.176	1100	0.0074	-
0.1776	1110	0.0019	-
0.1792	1120	0.0022	-
0.1808	1130	0.0063	-
0.1824	1140	0.0059	-
0.184	1150	0.0065	-
0.1856	1160	0.0151	-
0.1872	1170	0.0034	-
0.1888	1180	0.0033	-
0.1904	1190	0.0085	-
0.192	1200	0.0041	-
0.1936	1210	0.0084	-
0.1952	1220	0.004	-
0.1968	1230	0.0148	-
0.1984	1240	0.0111	-
0.2	1250	0.0125	-
0.2016	1260	0.0086	-
0.2032	1270	0.0042	-
0.2048	1280	0.0041	-
0.2064	1290	0.0078	-
0.208	1300	0.0042	-
0.2096	1310	0.0078	-
0.2112	1320	0.0065	-
0.2128	1330	0.0079	-
0.2144	1340	0.0157	-
0.216	1350	0.0086	-
0.2176	1360	0.0057	-
0.2192	1370	0.0025	-
0.2208	1380	0.0057	-
0.2224	1390	0.0051	-
0.224	1400	0.0054	-
0.2256	1410	0.0048	-
0.2272	1420	0.01	-
0.2288	1430	0.0087	-
0.2304	1440	0.0053	-
0.232	1450	0.0046	-
0.2336	1460	0.004	-
0.2352	1470	0.0062	-
0.2368	1480	0.0088	-
0.2384	1490	0.0093	-
0.24	1500	0.0005	-
0.2416	1510	0.0074	-
0.2432	1520	0.0042	-
0.2448	1530	0.0072	-
0.2464	1540	0.0007	-
0.248	1550	0.005	-
0.2496	1560	0.002	-
0.2512	1570	0.001	-
0.2528	1580	0.0062	-
0.2544	1590	0.0004	-
0.256	1600	0.0009	-
0.2576	1610	0.0041	-
0.2592	1620	0.0119	-
0.2608	1630	0.0011	-
0.2624	1640	0.0104	-
0.264	1650	0.0037	-
0.2656	1660	0.0005	-
0.2672	1670	0.004	-
0.2688	1680	0.0036	-
0.2704	1690	0.0037	-
0.272	1700	0.0013	-
0.2736	1710	0.0004	-
0.2752	1720	0.0006	-
0.2768	1730	0.0065	-
0.2784	1740	0.0033	-
0.28	1750	0.0009	-
0.2816	1760	0.0117	-
0.2832	1770	0.0033	-
0.2848	1780	0.0032	-
0.2864	1790	0.0037	-
0.288	1800	0.0022	-
0.2896	1810	0.0011	-
0.2912	1820	0.0006	-
0.2928	1830	0.0007	-
0.2944	1840	0.0054	-
0.296	1850	0.0007	-
0.2976	1860	0.0035	-
0.2992	1870	0.0038	-
0.3008	1880	0.0075	-
0.3024	1890	0.0017	-
0.304	1900	0.0005	-
0.3056	1910	0.0002	-
0.3072	1920	0.0002	-
0.3088	1930	0.0002	-
0.3104	1940	0.0033	-
0.312	1950	0.0085	-
0.3136	1960	0.0004	-
0.3152	1970	0.0005	-
0.3168	1980	0.0002	-
0.3184	1990	0.003	-
0.32	2000	0.0007	-
0.3216	2010	0.0009	-
0.3232	2020	0.0003	-
0.3248	2030	0.0012	-
0.3264	2040	0.0086	-
0.328	2050	0.001	-
0.3296	2060	0.0009	-
0.3312	2070	0.0029	-
0.3328	2080	0.0033	-
0.3344	2090	0.0005	-
0.336	2100	0.0003	-
0.3376	2110	0.0033	-
0.3392	2120	0.0029	-
0.3408	2130	0.0001	-
0.3424	2140	0.0057	-
0.344	2150	0.0001	-
0.3456	2160	0.0002	-
0.3472	2170	0.004	-
0.3488	2180	0.002	-
0.3504	2190	0.0073	-
0.352	2200	0.0004	-
0.3536	2210	0.0006	-
0.3552	2220	0.0004	-
0.3568	2230	0.0032	-
0.3584	2240	0.007	-
0.36	2250	0.0096	-
0.3616	2260	0.0032	-
0.3632	2270	0.0006	-
0.3648	2280	0.0002	-
0.3664	2290	0.0032	-
0.368	2300	0.0002	-
0.3696	2310	0.0025	-
0.3712	2320	0.0002	-
0.3728	2330	0.0053	-
0.3744	2340	0.0017	-
0.376	2350	0.0013	-
0.3776	2360	0.0001	-
0.3792	2370	0.0032	-
0.3808	2380	0.0002	-
0.3824	2390	0.0019	-
0.384	2400	0.0015	-
0.3856	2410	0.0009	-
0.3872	2420	0.0006	-
0.3888	2430	0.0032	-
0.3904	2440	0.0033	-
0.392	2450	0.0003	-
0.3936	2460	0.0003	-
0.3952	2470	0.0016	-
0.3968	2480	0.0065	-
0.3984	2490	0.0011	-
0.4	2500	0.0032	-
0.4016	2510	0.0045	-
0.4032	2520	0.0001	-
0.4048	2530	0.0004	-
0.4064	2540	0.0001	-
0.408	2550	0.0027	-
0.4096	2560	0.0032	-
0.4112	2570	0.0034	-
0.4128	2580	0.0057	-
0.4144	2590	0.0029	-
0.416	2600	0.0008	-
0.4176	2610	0.0002	-
0.4192	2620	0.0033	-
0.4208	2630	0.0004	-
0.4224	2640	0.0057	-
0.424	2650	0.0001	-
0.4256	2660	0.0048	-
0.4272	2670	0.0043	-
0.4288	2680	0.0011	-
0.4304	2690	0.0053	-
0.432	2700	0.0001	-
0.4336	2710	0.0045	-
0.4352	2720	0.0032	-
0.4368	2730	0.0034	-
0.4384	2740	0.0031	-
0.44	2750	0.0065	-
0.4416	2760	0.0013	-
0.4432	2770	0.0027	-
0.4448	2780	0.0014	-
0.4464	2790	0.0036	-
0.448	2800	0.0009	-
0.4496	2810	0.0053	-
0.4512	2820	0.0001	-
0.4528	2830	0.0005	-
0.4544	2840	0.0006	-
0.456	2850	0.0015	-
0.4576	2860	0.0028	-
0.4592	2870	0.0006	-
0.4608	2880	0.0001	-
0.4624	2890	0.0024	-
0.464	2900	0.0012	-
0.4656	2910	0.0003	-
0.4672	2920	0.0028	-
0.4688	2930	0.0022	-
0.4704	2940	0.0002	-
0.472	2950	0.0006	-
0.4736	2960	0.0002	-
0.4752	2970	0.0034	-
0.4768	2980	0.0032	-
0.4784	2990	0.0001	-
0.48	3000	0.0001	-
0.4816	3010	0.0003	-
0.4832	3020	0.0001	-
0.4848	3030	0.0011	-
0.4864	3040	0.0001	-
0.488	3050	0.0003	-
0.4896	3060	0.0031	-
0.4912	3070	0.0032	-
0.4928	3080	0.0028	-
0.4944	3090	0.0032	-
0.496	3100	0.0002	-
0.4976	3110	0.0001	-
0.4992	3120	0.0008	-
0.5008	3130	0.0028	-
0.5024	3140	0.0001	-
0.504	3150	0.0001	-
0.5056	3160	0.0001	-
0.5072	3170	0.0007	-
0.5088	3180	0.0054	-
0.5104	3190	0.0001	-
0.512	3200	0.0001	-
0.5136	3210	0.0001	-
0.5152	3220	0.0001	-
0.5168	3230	0.0027	-
0.5184	3240	0.0001	-
0.52	3250	0.0028	-
0.5216	3260	0.0001	-
0.5232	3270	0.0001	-
0.5248	3280	0.0007	-
0.5264	3290	0.0001	-
0.528	3300	0.0001	-
0.5296	3310	0.0001	-
0.5312	3320	0.0001	-
0.5328	3330	0.004	-
0.5344	3340	0.0001	-
0.536	3350	0.0049	-
0.5376	3360	0.0034	-
0.5392	3370	0.0004	-
0.5408	3380	0.0001	-
0.5424	3390	0.001	-
0.544	3400	0.0023	-
0.5456	3410	0.0019	-
0.5472	3420	0.0001	-
0.5488	3430	0.0027	-
0.5504	3440	0.0002	-
0.552	3450	0.0016	-
0.5536	3460	0.0001	-
0.5552	3470	0.0001	-
0.5568	3480	0.0005	-
0.5584	3490	0.0	-
0.56	3500	0.0001	-
0.5616	3510	0.0001	-
0.5632	3520	0.0001	-
0.5648	3530	0.0001	-
0.5664	3540	0.003	-
0.568	3550	0.0001	-
0.5696	3560	0.0002	-
0.5712	3570	0.0001	-
0.5728	3580	0.0001	-
0.5744	3590	0.0002	-
0.576	3600	0.0	-
0.5776	3610	0.0001	-
0.5792	3620	0.0034	-
0.5808	3630	0.0001	-
0.5824	3640	0.0001	-
0.584	3650	0.0001	-
0.5856	3660	0.0001	-
0.5872	3670	0.0003	-
0.5888	3680	0.0031	-
0.5904	3690	0.0001	-
0.592	3700	0.0001	-
0.5936	3710	0.003	-
0.5952	3720	0.0002	-
0.5968	3730	0.0031	-
0.5984	3740	0.0001	-
0.6	3750	0.0035	-
0.6016	3760	0.0001	-
0.6032	3770	0.003	-
0.6048	3780	0.0033	-
0.6064	3790	0.0026	-
0.608	3800	0.0024	-
0.6096	3810	0.0002	-
0.6112	3820	0.0001	-
0.6128	3830	0.0001	-
0.6144	3840	0.0001	-
0.616	3850	0.0001	-
0.6176	3860	0.0022	-
0.6192	3870	0.0001	-
0.6208	3880	0.0004	-
0.6224	3890	0.0066	-
0.624	3900	0.0033	-
0.6256	3910	0.0001	-
0.6272	3920	0.0001	-
0.6288	3930	0.0001	-
0.6304	3940	0.0032	-
0.632	3950	0.0003	-
0.6336	3960	0.0031	-
0.6352	3970	0.0001	-
0.6368	3980	0.0001	-
0.6384	3990	0.0001	-
0.64	4000	0.0001	-
0.6416	4010	0.0003	-
0.6432	4020	0.0001	-
0.6448	4030	0.0029	-
0.6464	4040	0.0001	-
0.648	4050	0.0001	-
0.6496	4060	0.0029	-
0.6512	4070	0.0001	-
0.6528	4080	0.0001	-
0.6544	4090	0.0001	-
0.656	4100	0.0001	-
0.6576	4110	0.0001	-
0.6592	4120	0.0001	-
0.6608	4130	0.0001	-
0.6624	4140	0.0001	-
0.664	4150	0.0001	-
0.6656	4160	0.0023	-
0.6672	4170	0.0002	-
0.6688	4180	0.0002	-
0.6704	4190	0.0014	-
0.672	4200	0.0004	-
0.6736	4210	0.0035	-
0.6752	4220	0.0001	-
0.6768	4230	0.0005	-
0.6784	4240	0.0001	-
0.68	4250	0.0029	-
0.6816	4260	0.0001	-
0.6832	4270	0.0001	-
0.6848	4280	0.0001	-
0.6864	4290	0.0001	-
0.688	4300	0.0003	-
0.6896	4310	0.0002	-
0.6912	4320	0.0001	-
0.6928	4330	0.0	-
0.6944	4340	0.0	-
0.696	4350	0.0	-
0.6976	4360	0.0001	-
0.6992	4370	0.0	-
0.7008	4380	0.0	-
0.7024	4390	0.0	-
0.704	4400	0.0	-
0.7056	4410	0.0	-
0.7072	4420	0.0	-
0.7088	4430	0.0	-
0.7104	4440	0.0001	-
0.712	4450	0.0001	-
0.7136	4460	0.0	-
0.7152	4470	0.0	-
0.7168	4480	0.0001	-
0.7184	4490	0.0	-
0.72	4500	0.0	-
0.7216	4510	0.0	-
0.7232	4520	0.0	-
0.7248	4530	0.0	-
0.7264	4540	0.0001	-
0.728	4550	0.0058	-
0.7296	4560	0.0001	-
0.7312	4570	0.0002	-
0.7328	4580	0.0001	-
0.7344	4590	0.0	-
0.736	4600	0.0001	-
0.7376	4610	0.0001	-
0.7392	4620	0.0	-
0.7408	4630	0.0002	-
0.7424	4640	0.0	-
0.744	4650	0.0	-
0.7456	4660	0.0004	-
0.7472	4670	0.0	-
0.7488	4680	0.0001	-
0.7504	4690	0.0	-
0.752	4700	0.0	-
0.7536	4710	0.0001	-
0.7552	4720	0.0001	-
0.7568	4730	0.0	-
0.7584	4740	0.0037	-
0.76	4750	0.0001	-
0.7616	4760	0.0032	-
0.7632	4770	0.0	-
0.7648	4780	0.0	-
0.7664	4790	0.0001	-
0.768	4800	0.0031	-
0.7696	4810	0.0001	-
0.7712	4820	0.0002	-
0.7728	4830	0.0	-
0.7744	4840	0.0001	-
0.776	4850	0.0001	-
0.7776	4860	0.0002	-
0.7792	4870	0.0	-
0.7808	4880	0.0	-
0.7824	4890	0.0001	-
0.784	4900	0.0	-
0.7856	4910	0.0	-
0.7872	4920	0.0001	-
0.7888	4930	0.0	-
0.7904	4940	0.0	-
0.792	4950	0.0001	-
0.7936	4960	0.0	-
0.7952	4970	0.0001	-
0.7968	4980	0.0	-
0.7984	4990	0.0029	-
0.8	5000	0.0001	-
0.8016	5010	0.0	-
0.8032	5020	0.0001	-
0.8048	5030	0.0005	-
0.8064	5040	0.0	-
0.808	5050	0.0	-
0.8096	5060	0.0014	-
0.8112	5070	0.0031	-
0.8128	5080	0.0	-
0.8144	5090	0.0001	-
0.816	5100	0.0	-
0.8176	5110	0.0001	-
0.8192	5120	0.0001	-
0.8208	5130	0.0	-
0.8224	5140	0.0	-
0.824	5150	0.0001	-
0.8256	5160	0.0	-
0.8272	5170	0.0	-
0.8288	5180	0.0	-
0.8304	5190	0.0006	-
0.832	5200	0.006	-
0.8336	5210	0.0032	-
0.8352	5220	0.0001	-
0.8368	5230	0.0	-
0.8384	5240	0.0	-
0.84	5250	0.0	-
0.8416	5260	0.0031	-
0.8432	5270	0.0001	-
0.8448	5280	0.0017	-
0.8464	5290	0.0009	-
0.848	5300	0.0001	-
0.8496	5310	0.0001	-
0.8512	5320	0.0004	-
0.8528	5330	0.0	-
0.8544	5340	0.003	-
0.856	5350	0.0002	-
0.8576	5360	0.0001	-
0.8592	5370	0.0001	-
0.8608	5380	0.0	-
0.8624	5390	0.0001	-
0.864	5400	0.0001	-
0.8656	5410	0.0	-
0.8672	5420	0.0	-
0.8688	5430	0.0001	-
0.8704	5440	0.0	-
0.872	5450	0.0	-
0.8736	5460	0.0	-
0.8752	5470	0.0001	-
0.8768	5480	0.0	-
0.8784	5490	0.0	-
0.88	5500	0.0	-
0.8816	5510	0.0001	-
0.8832	5520	0.0	-
0.8848	5530	0.0	-
0.8864	5540	0.0	-
0.888	5550	0.0031	-
0.8896	5560	0.0	-
0.8912	5570	0.0001	-
0.8928	5580	0.0	-
0.8944	5590	0.0	-
0.896	5600	0.0	-
0.8976	5610	0.0001	-
0.8992	5620	0.0	-
0.9008	5630	0.0002	-
0.9024	5640	0.0031	-
0.904	5650	0.0	-
0.9056	5660	0.0	-
0.9072	5670	0.0	-
0.9088	5680	0.0001	-
0.9104	5690	0.0	-
0.912	5700	0.0	-
0.9136	5710	0.0	-
0.9152	5720	0.0032	-
0.9168	5730	0.0001	-
0.9184	5740	0.0024	-
0.92	5750	0.0	-
0.9216	5760	0.0	-
0.9232	5770	0.0017	-
0.9248	5780	0.0	-
0.9264	5790	0.0001	-
0.928	5800	0.0001	-
0.9296	5810	0.0	-
0.9312	5820	0.0	-
0.9328	5830	0.0	-
0.9344	5840	0.0	-
0.936	5850	0.0	-
0.9376	5860	0.0031	-
0.9392	5870	0.0	-
0.9408	5880	0.0	-
0.9424	5890	0.0	-
0.944	5900	0.0031	-
0.9456	5910	0.0	-
0.9472	5920	0.0	-
0.9488	5930	0.0	-
0.9504	5940	0.0	-
0.952	5950	0.0	-
0.9536	5960	0.0001	-
0.9552	5970	0.0	-
0.9568	5980	0.0	-
0.9584	5990	0.0031	-
0.96	6000	0.0001	-
0.9616	6010	0.0	-
0.9632	6020	0.0	-
0.9648	6030	0.0	-
0.9664	6040	0.0	-
0.968	6050	0.0	-
0.9696	6060	0.0	-
0.9712	6070	0.0	-
0.9728	6080	0.0027	-
0.9744	6090	0.0	-
0.976	6100	0.0031	-
0.9776	6110	0.003	-
0.9792	6120	0.0	-
0.9808	6130	0.0	-
0.9824	6140	0.0	-
0.984	6150	0.0	-
0.9856	6160	0.0	-
0.9872	6170	0.0	-
0.9888	6180	0.0028	-
0.9904	6190	0.0	-
0.992	6200	0.0	-
0.9936	6210	0.0	-
0.9952	6220	0.0	-
0.9968	6230	0.0	-
0.9984	6240	0.0	-
1.0	6250	0.0	0.0479

Framework Versions

Python: 3.10.12
SetFit: 1.1.0
Sentence Transformers: 3.2.0
Transformers: 4.44.2
PyTorch: 2.4.1+cu121
Datasets: 3.0.1
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}