Post
700
๐๐
๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ป๐ด ๐๐ผ๐๐ฟ ๐๐ง๐ ๐ ๐๐ฒ๐ฏ๐ฝ๐ฎ๐ด๐ฒ๐ ๐๐ผ ๐บ๐ฎ๐ฟ๐ธ๐ฑ๐ผ๐๐ป ๐ถ๐ ๐ป๐ผ๐ ๐ฝ๐ผ๐๐๐ถ๐ฏ๐น๐ฒ ๐ฒ๐ป๐ฑ-๐๐ผ-๐ฒ๐ป๐ฑ ๐๐ถ๐๐ต ๐ฎ ๐๐ถ๐บ๐ฝ๐น๐ฒ ๐๐๐ ! ๐
Jina just released Reader-LM, that handles the whole pipeline of extracting markdown from HTML webpages.
A while ago, Jina had released a completely code-based deterministic program to do this extraction, based on some heuristics : e.g., โif the text is in a <p> tag, keep it, but if itโs hidden behind another, remove itโ.
๐ค But they received complaints from readers: some found it too detailed, other not enough, depending on the pages.
โก๏ธ So they decided, ๐บ๐ฎ๐๐ฏ๐ฒ ๐ต๐ฒ๐๐ฟ๐ถ๐๐๐ถ๐ฐ๐ ๐๐ฒ๐ฟ๐ฒ ๐ป๐ผ๐ ๐ฒ๐ป๐ผ๐๐ด๐ต: ๐ถ๐ป๐๐๐ฒ๐ฎ๐ฑ, ๐๐ต๐ฒ๐ ๐๐ฟ๐ถ๐ฒ๐ฑ ๐๐ผ ๐๐ฟ๐ฎ๐ถ๐ป ๐ฎ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฐ๐ผ๐บ๐ฝ๐น๐ฒ๐๐ฒ ๐ฒ๐ ๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป. This LLM does not need to be very strong,but it should handle a very long context: itโs a challenging, โshallow-but-wideโ architecture.
๐ง๐ฒ๐ฐ๐ต๐ป๐ถ๐ฐ๐ฎ๐น ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
2๏ธโฃ models: Reader-LM-0.5B and 1.5B
โ๏ธ Two stages of training: first, short and simple HTML to get the basics, then ramp up to longer and harder HTML up to 128k tokens
๐ Use contrastive search for decoding: this empirically reduces โrepeating outputโ issues
โก๏ธ Their models beat much larger models at HTML extraction ๐ฅ
๐ค Weights available on HF (sadly cc-by-nc license): jinaai/reader-lm-1.5b
Jina just released Reader-LM, that handles the whole pipeline of extracting markdown from HTML webpages.
A while ago, Jina had released a completely code-based deterministic program to do this extraction, based on some heuristics : e.g., โif the text is in a <p> tag, keep it, but if itโs hidden behind another, remove itโ.
๐ค But they received complaints from readers: some found it too detailed, other not enough, depending on the pages.
โก๏ธ So they decided, ๐บ๐ฎ๐๐ฏ๐ฒ ๐ต๐ฒ๐๐ฟ๐ถ๐๐๐ถ๐ฐ๐ ๐๐ฒ๐ฟ๐ฒ ๐ป๐ผ๐ ๐ฒ๐ป๐ผ๐๐ด๐ต: ๐ถ๐ป๐๐๐ฒ๐ฎ๐ฑ, ๐๐ต๐ฒ๐ ๐๐ฟ๐ถ๐ฒ๐ฑ ๐๐ผ ๐๐ฟ๐ฎ๐ถ๐ป ๐ฎ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฐ๐ผ๐บ๐ฝ๐น๐ฒ๐๐ฒ ๐ฒ๐ ๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป. This LLM does not need to be very strong,but it should handle a very long context: itโs a challenging, โshallow-but-wideโ architecture.
๐ง๐ฒ๐ฐ๐ต๐ป๐ถ๐ฐ๐ฎ๐น ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
2๏ธโฃ models: Reader-LM-0.5B and 1.5B
โ๏ธ Two stages of training: first, short and simple HTML to get the basics, then ramp up to longer and harder HTML up to 128k tokens
๐ Use contrastive search for decoding: this empirically reduces โrepeating outputโ issues
โก๏ธ Their models beat much larger models at HTML extraction ๐ฅ
๐ค Weights available on HF (sadly cc-by-nc license): jinaai/reader-lm-1.5b
