Tonic commited on
Commit
4dda336
โ€ข
1 Parent(s): 6aed56b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +80 -0
README.md CHANGED
@@ -7,6 +7,86 @@ language:
7
  library_name: adapter-transformers
8
  ---
9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  # Model Card for K23 MiniMed
11
 
12
  This is a Mistral 7b Beta Medical Fine Tune with a short number of steps , inspired by [Wonhyeong Seo](https://www.huggingface.co/wseo) great mentorship during Krew x Huggingface 2023 hackathon.
 
7
  library_name: adapter-transformers
8
  ---
9
 
10
+ # K23 MiniMed ๋ชจ๋ธ ์นด๋“œ
11
+
12
+ K23 MiniMed๋Š” Krew x Huggingface 2023 ํ•ด์ปคํ†ค์—์„œ ์›ํ˜•์„ ๋ฉ˜ํ† ์˜ ์ง€๋„ํ•˜์— ๊ฐœ๋ฐœ๋œ Mistral 7b Beta Medical Fine Tune ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.
13
+
14
+ ## ๋ชจ๋ธ ์„ธ๋ถ€์‚ฌํ•ญ
15
+
16
+ - **๊ฐœ๋ฐœ์ž:** [Tonic](https://huggingface.co/Tonic)
17
+
18
+ - **ํ›„์›:** [Tonic](https://huggingface.co/Tonic)
19
+ - **๊ณต์œ ์ž:** K23-Krew-Hackathon
20
+ - **๋ชจ๋ธ ์œ ํ˜•:** Mistral 7B-Beta Medical Fine Tune
21
+ - **์–ธ์–ด (NLP):** ์˜์–ด
22
+ - **๋ผ์ด์„ผ์Šค:** MIT
23
+ - **Fine-tuning ๊ธฐ๋ฐ˜ ๋ชจ๋ธ:** [Zephyr 7B-Beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)
24
+
25
+ ### ๋ชจ๋ธ ์ถœ์ฒ˜
26
+ - **์ €์žฅ์†Œ:** [github](https://github.com/Josephrp/AI-challenge-hackathon/blob/master/mistral7b-beta_finetune.ipynb)
27
+ - **๋ฐ๋ชจ:** [pseudolab/K23MiniMed](https://huggingface.co/spaces/pseudolab/K23MiniMed)
28
+ ## ์‚ฌ์šฉ๋ฒ•
29
+ ์ด ๋ชจ๋ธ์€ ๊ต์œก ๋ชฉ์ ์œผ๋กœ๋งŒ ์˜ํ•™ ์งˆ๋ฌธ ๋‹ต๋ณ€์„ ์œ„ํ•œ ๋Œ€ํ™”ํ˜• ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์šฉ์ž…๋‹ˆ๋‹ค.
30
+ ### ์ง์ ‘ ์‚ฌ์šฉ
31
+
32
+ Gradio ์ฑ—๋ด‡ ์•ฑ์„ ๋งŒ๋“ค์–ด ์˜ํ•™์  ์งˆ๋ฌธ์„ ํ•˜๊ณ  ๋Œ€ํ™”์‹์œผ๋กœ ๋‹ต๋ณ€์„ ๋ฐ›์Šต๋‹ˆ๋‹ค.
33
+
34
+ ### ํ•˜๋ฅ˜ ์‚ฌ์šฉ
35
+
36
+ ์ด ๋ชจ๋ธ์€ ๊ต์œก์šฉ์œผ๋กœ๋งŒ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค. ์ถ”๊ฐ€์ ์ธ Fine-tuning๊ณผ ์‚ฌ์šฉ ์˜ˆ์‹œ๋กœ๋Š” ๊ณต์ค‘ ๋ณด๊ฑด & ์œ„์ƒ, ๊ฐœ์ธ ๋ณด๊ฑด & ์œ„์ƒ, ์˜ํ•™ Q & A๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
37
+
38
+ ### ์ถ”์ฒœ์‚ฌํ•ญ
39
+
40
+ ์‚ฌ์šฉ ์ „์— ํ•ญ์ƒ ์ด ๋ชจ๋ธ์„ ํ‰๊ฐ€ํ•˜๊ณ  ๋ฒค์น˜๋งˆํ‚นํ•˜์‹ญ์‹œ์˜ค. ์‚ฌ์šฉ ์ „์— ํŽธํ–ฅ์„ ํ‰๊ฐ€ํ•˜์‹ญ์‹œ์˜ค. ๊ทธ๋Œ€๋กœ ์‚ฌ์šฉํ•˜์ง€ ๋งˆ์‹œ๊ณ  ์ถ”๊ฐ€์ ์œผ๋กœ Fine-tuningํ•˜์‹ญ์‹œ์˜ค.
41
+
42
+ ## ํ›ˆ๋ จ ์„ธ๋ถ€์‚ฌํ•ญ
43
+
44
+ ๋ชจ๋ธ์˜ ํ›ˆ๋ จ ์†์‹ค์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค:
45
+ | ๋‹จ๊ณ„ | ํ›ˆ๋ จ ์†์‹ค |
46
+ |------|--------------|
47
+ | 50 | 0.993800 |
48
+ | 100 | 0.620600 |
49
+ | 150 | 0.547100 |
50
+ | 200 | 0.524100 |
51
+ | 250 | 0.520500 |
52
+ | 300 | 0.559800 |
53
+ | 350 | 0.535500 |
54
+ | 400 | 0.505400 |
55
+ ### ํ›ˆ๋ จ ๋ฐ์ดํ„ฐ
56
+ ๋ชจ๋ธ์˜ ํ•™์Šต ๊ฐ€๋Šฅํ•œ ๋งค๊ฐœ๋ณ€์ˆ˜: 21260288, ๋ชจ๋“  ๋งค๊ฐœ๋ณ€์ˆ˜: 3773331456, ํ•™์Šต ๊ฐ€๋Šฅํ•œ %: 0.5634354746703705.
57
+
58
+ ### ๊ฒฐ๊ณผ
59
+
60
+ global_step=400์—์„œ์˜ ํ›ˆ๋ จ ์†์‹ค์€ 0.6008514881134033์ž…๋‹ˆ๋‹ค.
61
+
62
+ ## ํ™˜๊ฒฝ ์˜ํ–ฅ
63
+
64
+ ๋ชจ๋ธ์˜ ํ™˜๊ฒฝ ์˜ํ–ฅ์€ ๋จธ์‹ ๋Ÿฌ๋‹ ์˜ํ–ฅ ๊ณ„์‚ฐ๊ธฐ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๊ณ„์‚ฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ถ”์ •์„ ์ œ๊ณตํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” ๋” ๋งŽ์€ ์„ธ๋ถ€ ์ •๋ณด๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
65
+
66
+ ## ๊ธฐ์ˆ  ์‚ฌ์–‘
67
+
68
+ ### ๋ชจ๋ธ ์•„ํ‚คํ…์ฒ˜์™€ ๋ชฉํ‘œ
69
+
70
+ ๋ชจ๋ธ์€ ํŠน์ • ์„ค์ •์„ ๊ฐ€์ง„ PeftModelForCausalLM์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
71
+
72
+ ### ์ปดํ“จํŒ… ์ธํ”„๋ผ
73
+
74
+ #### ํ•˜๋“œ์›จ์–ด
75
+
76
+ ๋ชจ๋ธ์€ A100 ํ•˜๋“œ์›จ์–ด์—์„œ ํ›ˆ๋ จ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
77
+
78
+ #### ์†Œํ”„ํŠธ์›จ์–ด
79
+
80
+ ์‚ฌ์šฉ๋œ ์†Œํ”„ํŠธ์›จ์–ด์—๋Š” peft, torch, bitsandbytes, python, ๊ทธ๋ฆฌ๊ณ  huggingface๊ฐ€ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค.
81
+
82
+ ## ๋ชจ๋ธ ์นด๋“œ ์ž‘์„ฑ์ž
83
+
84
+ [Tonic](https://huggingface.co/Tonic)
85
+
86
+ ## ๋ชจ๋ธ ์นด๋“œ ์—ฐ๋ฝ์ฒ˜
87
+
88
+ [Tonic](https://huggingface.co/Tonic)
89
+
90
  # Model Card for K23 MiniMed
91
 
92
  This is a Mistral 7b Beta Medical Fine Tune with a short number of steps , inspired by [Wonhyeong Seo](https://www.huggingface.co/wseo) great mentorship during Krew x Huggingface 2023 hackathon.