Spaces:

gufett0
/

chatbot-llamaindex

Sleeping

gufett0 commited on Sep 18

Commit

f473ccd

•

1 Parent(s): 63c9ed5

HuggingFaceLLM

Files changed (1) hide show

backend.py CHANGED Viewed

@@ -66,7 +66,7 @@ llm = HuggingFaceLLM(
     model_name=model_id,
     device_map="auto",
     # change these settings below depending on your GPU
-    model_kwargs={"torch_dtype": torch.float16, "load_in_8bit": True},
 )
 #Settings.llm = GemmaLLMInterface()

     model_name=model_id,
     device_map="auto",
     # change these settings below depending on your GPU
+    model_kwargs={"torch_dtype": torch.bfloat16, "load_in_8bit": True},
 )
 #Settings.llm = GemmaLLMInterface()