nuprl/MultiPL-T-starcoderbase1b-racket-selfinstruct

Model Card for StarCoderBase1B-Racket-SelfInstruct

Each commit to this repository has a checkpoint (one per epoch) for a fine-tuned StarCoderBase-1B. The dataset for fine-tuning is a Racket self-instruction dataset. As shown in Evaluation below, self-instruction was not effective, and this model is barely any better than StarCoderBase-1B.

Finetuning Dataset and Hyperparameters

• Dataset: https://huggingface.co/datasets/nuprl/MultiPL-T-racket-selfinstruct
• Base Model: https://huggingface.co/bigcode/starcoderbase-1b
• Batch size: 8
• Epochs: 7
• Learning rate: 3e-05
• Warmup steps: 100
• Sequence length: 2048

Evaluation

The results on MultiPL-HumanEval-Racket are as follows:

Dataset,Pass@k,Estimate,NumProblems,MinCompletions,MaxCompletions
humaneval-rkt-checkpoint_1494-0.2-reworded,1,7.70,161,50,50
humaneval-rkt-checkpoint_1992-0.2-reworded,1,6.86,161,50,50
humaneval-rkt-checkpoint_2490-0.2-reworded,1,6.82,161,50,50
humaneval-rkt-checkpoint_2988-0.2-reworded,1,6.91,161,50,50
humaneval-rkt-checkpoint_498-0.2-reworded,1,6.19,161,50,50
humaneval-rkt-checkpoint_6973-0.2-reworded,1,6.53,161,50,50
humaneval-rkt-checkpoint_996-0.2-reworded,1,7.08,161,50,50