finetunedPHP_starcoder2
This model is a fine-tuned version of bigcode/starcoder2-3b on bigcode/the-stack-smol.
Model description
The finetunedPHP_starcoder2
model is based on the starcoder2-3b
architecture, fine-tuned specifically on PHP code from the-stack-smol
dataset. It is intended for code generation tasks related to PHP programming.
Intended uses & limitations
The finetunedPHP_starcoder2
model is suitable for generating PHP code snippets for various purposes, including code completion, syntax suggestions, and code generation tasks. However, it may have limitations in generating complex or domain-specific code, and users should verify the generated code for correctness and security.
Training and evaluation data
The model was trained on a dataset consisting of PHP code samples collected from the-stack-smol
dataset. The training data included code snippets from PHP repositories, forums, and online tutorials.
Training procedure
1. Data and Model Preparation:
- Load the PHP dataset from my repository
bigcode/the-stack-smol
. - Extract the relevant PHP data
data/php
samples for training. - Utilize the
starcoder2-3b
model pre-trained on a diverse range of programming languages, including PHP, from the Hugging Face Hub. - Ensure the model is configured with '4-bit' quantization for efficient computation.
2. Data Processing:
- Tokenize the PHP code snippets using the model's tokenizer.
- Clean the code by removing comments and normalizing indentation.
- Prepare input examples suitable for the model, considering its architecture and objectives.
3. Training Configuration:
- Initialize a Trainer object for fine-tuning, leveraging the Transformers library.
- Define training parameters, including:
- Learning rate, optimizer, and scheduler settings.
- Gradient accumulation steps to balance memory usage.
- Loss function, typically cross-entropy for language modeling.
- Metrics for evaluating model performance.
- Specify GPU utilization for accelerated training.
- Handle potential distributed training with multiple processes.
4. Model Training:
- Commence training for a specified number of steps.
- Iterate through batches of preprocessed PHP code examples.
- Feed examples into the model and compute predictions.
- Calculate loss based on predicted and actual outcomes.
- Update model weights by backpropagating gradients through the network.
5. Evaluation (Optional):
- Periodically assess the model's performance on a validation set.
- Measure key metrics such as code completion accuracy or perplexity.
- Monitor training progress to fine-tune hyperparameters if necessary.
- Use wandb metric monitoring for live monitoring.
6. Save the Fine-tuned Model:
- Store the optimized model weights and configuration in the designated
output_dir
.
7. Model Sharing (Optional):
- Optionally, create a model card documenting the fine-tuning process and model specifications.
- Share the finetunedPHP_starcoder2 model on the Hugging Face Hub for broader accessibility and collaboration.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 8
- seed: 0
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 1000
- mixed_precision_training: Native AMP
Training results
Training results and performance metrics are present in the repo.
Framework versions
- PEFT 0.8.2
- Transformers 4.40.0.dev0
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2
- Downloads last month
- 7
Model tree for CodeHima/finetunedPHP_starcoder2
Base model
bigcode/starcoder2-3b