Appreciate the model drop!
But why is it only 4k? Its 2024 man, those are rookie numbers.
We will release the long context version soon, please stay tuned!
Is there a way to run it with Pytorch 2.0.0?
Got Error:
[nn.LayerNorm(dim, eps=eps, bias=bias) for _ in range(self.num_heads)]
File "../huggingface/modules/transformers_modules/stabilityai/stablelm-2-12b-chat/e2a306c50ac1900c408cc33c058ae48a33820cf1/modeling_stablelm.py", line 256, in
[nn.LayerNorm(dim, eps=eps, bias=bias) for _ in range(self.num_heads)]
TypeError: init() got an unexpected keyword argument 'bias'
Line 256:
[nn.LayerNorm(dim, eps=eps, bias=bias) for _ in range(self.num_heads)]
See: https://pytorch.org/docs/2.0/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm
torch.nn.LayerNorm(normalized_shape, eps=1e-05, elementwise_affine=True, device=None, dtype=None)
Parameter bias is missing.
