tsunemoto commited on
Commit
8ab6428
1 Parent(s): 16d342a

Upload folder using huggingface_hub

Browse files
.gitattributes CHANGED
@@ -33,3 +33,17 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ heimer-ipo-tinyllama-1.1b.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
37
+ heimer-ipo-tinyllama-1.1b.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
38
+ heimer-ipo-tinyllama-1.1b.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
39
+ heimer-ipo-tinyllama-1.1b.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
40
+ heimer-ipo-tinyllama-1.1b.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
41
+ heimer-ipo-tinyllama-1.1b.Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
42
+ heimer-ipo-tinyllama-1.1b.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
43
+ heimer-ipo-tinyllama-1.1b.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
44
+ heimer-ipo-tinyllama-1.1b.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
45
+ heimer-ipo-tinyllama-1.1b.Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
46
+ heimer-ipo-tinyllama-1.1b.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
47
+ heimer-ipo-tinyllama-1.1b.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
48
+ heimer-ipo-tinyllama-1.1b.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
49
+ heimer-ipo-tinyllama-1.1b.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: "Heimer-ipo-TinyLlama-1.1B Quantized in GGUF"
3
+ tags:
4
+ - GGUF
5
+ language: en
6
+ ---
7
+ ![Image description](https://i.postimg.cc/MGwhtFfF/tsune-fixed.png)
8
+
9
+ # Tsunemoto GGUF's of Heimer-ipo-TinyLlama-1.1B
10
+
11
+ This is a GGUF quantization of Heimer-ipo-TinyLlama-1.1B.
12
+
13
+ ## Original Repo Link:
14
+ [Original Repository](https://huggingface.co/abideen/Heimer-ipo-TinyLlama-1.1B)
15
+
16
+ ## Original Model Card:
17
+ ---
18
+
19
+ # Heimer-ipo-TinyLlama-1.1B
20
+
21
+
22
+ ![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/64e380b2e12618b261fa6ba0/a7joKICVpqGElN3mh2MpS.jpeg)
23
+
24
+
25
+ # WandB Experiment Tracking
26
+
27
+ Check out the experiment details in this [report](https://api.wandb.ai/links/zaiinn440/dqlt70dc)
28
+
29
+
30
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64e380b2e12618b261fa6ba0/HLhrSUFD1-e6f31F3WGK2.png)
31
+
32
+ # 🧩 IPO adaptation hyperparameters
33
+
34
+ ## LoRA:
35
+
36
+ r=8
37
+
38
+ lora_alpha=16
39
+
40
+ lora_dropout=0.05
41
+
42
+ bias="none"
43
+
44
+ task_type="CAUSAL_LM"
45
+
46
+ target_modules=['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
47
+
48
+ ## Training arguments:
49
+
50
+ per_device_train_batch_size=2
51
+
52
+ gradient_accumulation_steps=4
53
+
54
+ gradient_checkpointing=True
55
+
56
+ learning_rate=5e-5
57
+
58
+ lr_scheduler_type="cosine"
59
+
60
+ max_steps=50
61
+
62
+ optim="paged_adamw_32bit"
63
+
64
+ warmup_steps=10
65
+
66
+ ## DPOTrainer:
67
+
68
+ beta=0.1
69
+
70
+ max_prompt_length=1024
71
+
72
+ max_length=1536
73
+
74
+ loss="ipo"
75
+
76
+
77
+ ## 💻 Usage
78
+
79
+ Here's a [Colab notebook](https://colab.research.google.com/drive/11KEX1LG3nRBoeGR0Iyy-459XllGlLOA9?usp=sharing) to run Heimer-TinyLLama-1.1B in 4-bit precision on a free T4 GPU.
80
+
81
+ ```python
82
+ !pip install -qU transformers accelerate
83
+
84
+ from transformers import AutoTokenizer
85
+ import transformers
86
+ import torch
87
+
88
+ model = "abideen/Heimer-ipo-TinyLlama-1.1B"
89
+ messages = [{"role": "user", "content": "Explain what is Data science."}]
90
+
91
+ tokenizer = AutoTokenizer.from_pretrained(model)
92
+ prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
93
+ pipeline = transformers.pipeline(
94
+ "text-generation",
95
+ model=model,
96
+ torch_dtype=torch.float16,
97
+ device_map="auto",
98
+ )
99
+
100
+ outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
101
+ print(outputs[0]["generated_text"])
102
+ ```
103
+
104
+ "What is Data Science?
105
+ A data scientist is an individual who has a passion for data and knowledge of the technology that can be used to help make sense of data. Data scientists are often involved in the development of new software and software platforms, as well as analyzing and interpreting data.
106
+ What are the Important components of Data Science?
107
+ 1. Data: The data is the most important component of a data science project. Data science is the application of data science to make sense of data. Data scientists usually work with data, but data scientists are not necessarily data scientists.
108
+ 2. Analysis: This is the process of taking data and turning it into something useful.
109
+ 3. Modeling: The use of machine learning and statistical techniques.
110
+ 4. Prediction: The prediction of a future event, such as the future market share of a product or the future population of an area.
111
+ 5. Visualization: Displaying the data in a graphical or interactive format.
112
+ 6. Statistics: The use of statistical analysis techniques.
113
+ What are the Advantages of Data Science?
114
+ Data science is the application of data science to make sense of data."
heimer-ipo-tinyllama-1.1b.Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8780c6baf458eb041c6cd4b3b80e088dcf09a424b2fb377f90c881614bb31741
3
+ size 483115904
heimer-ipo-tinyllama-1.1b.Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88b1ecfcdc0f545bb2b23bd97773c615f10f0f01b7675fa3acb4c211bbfaeabe
3
+ size 592499584
heimer-ipo-tinyllama-1.1b.Q3_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:374779b7dff12cf7df88582f0ac7650f40d7c32573a9823d845c2cf3752ec4cf
3
+ size 550818688
heimer-ipo-tinyllama-1.1b.Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c46a40b2252361dc1ba9758c9fa1f64111e4532c97bb3d0ec9312f4b029f90d
3
+ size 500315008
heimer-ipo-tinyllama-1.1b.Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42836cbf9bc61d293bb79efe046f80cb71d62908fc4fdc4cc11d1d3e30061abe
3
+ size 637698944
heimer-ipo-tinyllama-1.1b.Q4_1.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef472f659fe552e6d346d3994c30514490491383f1092af2a80914378635fd0b
3
+ size 702350208
heimer-ipo-tinyllama-1.1b.Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18d7c2338bd1a30e67440c24df0ccbe411577e44e1ace8a7597945c4b197d58a
3
+ size 668787584
heimer-ipo-tinyllama-1.1b.Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1f360c71dc977be983c98397a581533cf3cde332496c8039c211fd8be307f035
3
+ size 643728256
heimer-ipo-tinyllama-1.1b.Q5_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0e4e362afe33e1211a26770d99ce56cf26c112b957345c5e488f214b6042865
3
+ size 767001472
heimer-ipo-tinyllama-1.1b.Q5_1.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:354410f12c0e6f6439fd1ee974dd97ad1e97eba61b84ea5f2d523860e69f55f4
3
+ size 831652736
heimer-ipo-tinyllama-1.1b.Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:655ce743e7c9c3a56671b1cbd9641a3656c29b2d051f73a83c0619f297a205cc
3
+ size 783016832
heimer-ipo-tinyllama-1.1b.Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a7c59ca136ac6b83e908e31a94916a1c6e33ce66e1891983bca3cd93be85016f
3
+ size 767001472
heimer-ipo-tinyllama-1.1b.Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3209a6df82e89fe1fc168739fca8ba3764f927a34dab788711767cbe4a4b65bd
3
+ size 904385408
heimer-ipo-tinyllama-1.1b.Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:496b95df77a9d2f62c240736e8936b1cecde875530aecc0cb826e636230d08ea
3
+ size 1170781056