Paragraph Breaks
#4
by
Shasho423
- opened
For me, in oobabooga, it is inserting "<0x0A><0x0A>" anywhere there should be a paragraph break, but it is not appearing in the chat as paragraph breaks, only a continuous single paragraph with "<0x0A><0x0A>" appearing occasionally between sentences. Any ideas?
I also saw similar discussions here: https://x.com/HenkPoley/status/1729347141695385736?s=20. It seems that this token is supposed to be new line "\n", but somehow decoded wrongly?
I guess it's because we forgot to include tokenizer files from openchat 3.5 according to this https://huggingface.co/TheBloke/Starling-LM-7B-alpha-GGUF/discussions/1 I just uploaded the relevant files.
Oh, yeah, I am actually using TheBloke's haha. Thank you!