I designing a Vector Database to work with Falcon 180B as the knowledge domain data, which tokenizer format is used? BPE, Wordpiece, Sentence piece? it is important as to match both.
· Sign up or log in to comment
