|
--- |
|
license: cc-by-4.0 |
|
datasets: |
|
- Aiwensile2/Minecraft_QA-pairs_Instruction_Dataset |
|
--- |
|
# MineMA: Fine-Tuned Models for Minecraft Q&A |
|
|
|
## Overview |
|
|
|
In this repository, we present the MineMA series of models, fine-tuned specifically for Minecraft-related Q&A tasks. Utilizing the LoRA method for efficient model fine-tuning, we have adapted pre-trained LLaMA models to respond accurately and effectively to Minecraft-related instructions and queries. Our fine-tuning process leverages the specially generated Minecraft dataset to ensure relevance and accuracy in the Q&A responses. |
|
|
|
## Models |
|
|
|
The MineMA series includes several models fine-tuned on different base models from the LLaMA series. Below is the list of the fine-tuned models provided in this repository: |
|
|
|
- **MineMA-8B**(v1, v2, v3, v4), derived from the base model LLaMA-3-8B-Instruct. |
|
- **MineMA-13B**(v1, v2), derived from the base model LLaMA-2-13B-Chat. |
|
- **MineMA-70B**, derived from the base model LLaMA-3-70B-Instruct. |
|
|
|
These models have been fine-tuned by using the **Minecraft_QA-pairs_Instruction_Dataset**. We have only released four models of MineMA-8B for the time being, and we will supplement more models in the future. |
|
|
|
## Fine-Tuning Methodology |
|
|
|
### LoRA Method for Fine-Tuning |
|
|
|
We employed the **LoRA (Low-Rank Adaptation)** method for fine-tuning our models. LoRA is a parameter-efficient training technique that introduces small, trainable low-rank matrices to adapt a pre-trained neural network, allowing for targeted updates without the need for retraining the entire model. This method strikes a balance between computational efficiency and training effectiveness. |
|
|
|
### Training Parameters |
|
|
|
Here are the specific training parameters: |
|
|
|
| Model | lora\_r | lora\_alpha | lora\_dropout | learning\_rate | weight\_decay | Single Round? | |
|
|--------------|---------|-------------|---------------|----------------|---------------|---------------| |
|
| MineMA-13B-v1| 64 | 128 | 0.1 | 1E-04 | 1E-04 | False | |
|
| MineMA-13B-v2| 128 | 256 | 0.1 | 1E-04 | 1E-04 | False | |
|
| MineMA-8B-v1 | 64 | 128 | 0.1 | 1E-04 | 1E-04 | True | |
|
| MineMA-8B-v2 | 32 | 64 | 0.1 | 1E-04 | 1E-04 | False | |
|
| MineMA-8B-v3 | 64 | 128 | 0.1 | 1E-04 | 1E-04 | False | |
|
| MineMA-8B-v4 | 128 | 256 | 0.1 | 1E-04 | 1E-04 | False | |
|
| MineMA-70B | 16 | 32 | 0.1 | 1E-04 | 1E-04 | True | |
|
|
|
## Dataset |
|
|
|
We used the **Minecraft_QA-pairs_Instruction_Dataset** for fine-tuning all the models in the MineMA series. This dataset has 390,317 instruction entries specifically designed for Minecraft-related Q&A tasks. You can access the dataset via the following link: |
|
|
|
[Minecraft_QA-pairs_Instruction_Dataset](https://huggingface.co/datasets/Aiwensile2/Minecraft_QA-pairs_Instruction_Dataset) |
|
|
|
## Details |
|
### License |
|
These models are made available under the Creative Commons Attribution 4.0 International License. |
|
|
|
### DOI |
|
10.57967/hf/2488 |
|
|
