license: cc-by-4.0
datasets:
- Aiwensile2/Minecraft_QA-pairs_Instruction_Dataset
MineMA: Fine-Tuned Models for Minecraft Q&A
Overview
In this repository, we present the MineMA series of models, fine-tuned specifically for Minecraft-related Q&A tasks. Utilizing the LoRA method for efficient model fine-tuning, we have adapted pre-trained LLaMA models to respond accurately and effectively to Minecraft-related instructions and queries. Our fine-tuning process leverages the specially generated Minecraft dataset to ensure relevance and accuracy in the Q&A responses.
Models
The MineMA series includes several models fine-tuned on different base models from the LLaMA series. Below is the list of the fine-tuned models provided in this repository:
- MineMA-8B(v1, v2, v3, v4), derived from the base model LLaMA-3-8B-Instruct.
- MineMA-13B(v1, v2), derived from the base model LLaMA-2-13B-Chat.
- MineMA-70B, derived from the base model LLaMA-3-70B-Instruct.
These models have been fine-tuned by using the Minecraft_QA-pairs_Instruction_Dataset. We have only released four models of MineMA-8B for the time being, and we will supplement more models in the future.
Fine-Tuning Methodology
LoRA Method for Fine-Tuning
We employed the LoRA (Low-Rank Adaptation) method for fine-tuning our models. LoRA is a parameter-efficient training technique that introduces small, trainable low-rank matrices to adapt a pre-trained neural network, allowing for targeted updates without the need for retraining the entire model. This method strikes a balance between computational efficiency and training effectiveness.
Training Parameters
Here are the specific training parameters:
| Model | lora_r | lora_alpha | lora_dropout | learning_rate | weight_decay | Single Round? |
|---|---|---|---|---|---|---|
| MineMA-13B-v1 | 64 | 128 | 0.1 | 1E-04 | 1E-04 | False |
| MineMA-13B-v2 | 128 | 256 | 0.1 | 1E-04 | 1E-04 | False |
| MineMA-8B-v1 | 64 | 128 | 0.1 | 1E-04 | 1E-04 | True |
| MineMA-8B-v2 | 32 | 64 | 0.1 | 1E-04 | 1E-04 | False |
| MineMA-8B-v3 | 64 | 128 | 0.1 | 1E-04 | 1E-04 | False |
| MineMA-8B-v4 | 128 | 256 | 0.1 | 1E-04 | 1E-04 | False |
| MineMA-70B | 16 | 32 | 0.1 | 1E-04 | 1E-04 | True |
Dataset
We used the Minecraft_QA-pairs_Instruction_Dataset for fine-tuning all the models in the MineMA series. This dataset has 390,317 instruction entries specifically designed for Minecraft-related Q&A tasks. You can access the dataset via the following link: