Merge your base model with your LoRA
Checkout a RAG system i made for local use:https://github.com/Tech-ware/ChatHX
Image to Prompt using fientuned PaliGemma for SD3