Randomness of the output of the trained model
#68
by
Sam1989
- opened
I optimized a Gemma-2b model for a classification task in 4-bit quantization with LoRA. It shows remarkably good results on test data. The model has been saved. When I reload the model and make predictions, I notice that the probabilities for the classes change slightly after each prediction. I let the model predict the same text 5 times in a row and got the following values for the best class:
[0.778,0.746,0.732,0.693,0.712]
Where does this randomness come from? It should be the same every prediction ? Because the weights and biases of the model are no longer changed .