| license: apache-2.0 | |
| # fasttext cbow on dclm400 | |
| A continuous-bag-of-words model trained on https://huggingface.co/datasets/mlfoundations/dclm-pool-400m-1x | |
| the cbow model was trained with https://github.com/facebookresearch/fastText/ | |
| the dataset was downloaded with git-lfs | |
| the dataset commit was: f20ae752116ce7b4ab15d31e1e40b094229bf911 | |
| the files decompressed with: | |
| `parallel "zstd --keep --stdout -d {} | jq .text > {/}.txt" ::: /root/lfs/dclm-pool-400m-1x/*.jsonl.zst` | |
| concatenated with | |
| `cat *.txt > CC_SHARD_ALL.jsonl.txt` | |
| the `sha256sum CC_SHARD_ALL.json.txt` is | |
| `576e4e79e76b9ca24dc77a8da0df17ad5efc9c5ca16c9a86f62e7b7b4ae8c640 CC_SHARD_ALL.jsonl.txt` | |
| then the fasttext model was trained with defaults settings from | |
| compiled with gcc 13.3.1 | |
| fasttext-repo (main branch) with the commit hash `1142dc4c4ecbc19cc16eee5cdd28472e689267e6` | |
| training command: | |
| `prlimit -m 3200000000 fasttext cbow -input CC_SHARD_ALL.jsonl.txt -output fasttext_models/model` | |
| the exact fasttext binary is included in this repo as `fasttext` | |
| the decompression and concatenating took a few hours. | |
| the model training took 100 hours on 8 cores plus a few hours to read in the words (fasttext) | |