add fasttext model, fasttext binary and readme

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model1.vec filter=lfs diff=lfs merge=lfs -text
+model1.bin filter=lfs diff=lfs merge=lfs -text
+fasttext filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,41 @@
 ---
 license: apache-2.0
 ---
+
+# fasttext cbow on dclm400
+
+A continuous-bag-of-words model trained on https://huggingface.co/datasets/mlfoundations/dclm-pool-400m-1x
+
+the cbow model was trained with https://github.com/facebookresearch/fastText/
+
+the dataset was downloaded with git-lfs
+
+the dataset commit was: f20ae752116ce7b4ab15d31e1e40b094229bf911
+
+the files decompressed with:
+
+`parallel "zstd --keep --stdout -d {} | jq .text > {/}.txt" ::: /root/lfs/dclm-pool-400m-1x/*.jsonl.zst`
+
+concatenated with
+
+`cat *.txt > CC_SHARD_ALL.jsonl.txt`
+
+the `sha256sum CC_SHARD_ALL.json.txt` is    
+
+`576e4e79e76b9ca24dc77a8da0df17ad5efc9c5ca16c9a86f62e7b7b4ae8c640  CC_SHARD_ALL.jsonl.txt`
+    
+then the fasttext model was trained with defaults settings from
+
+compiled with gcc 13.3.1
+
+fasttext-repo (main branch) with the commit hash `1142dc4c4ecbc19cc16eee5cdd28472e689267e6`
+
+training command:
+
+`prlimit -m 3200000000 fasttext cbow -input CC_SHARD_ALL.jsonl.txt -output fasttext_models/model`
+
+the exact fasttext binary is included in this repo as `fasttext`
+    
+the decompression and concatenating took a few hours.
+
+the model training took 100 hours on 8 cores plus a few hours to read in the words (fasttext)

fasttext

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2eb67f89e29c7da08b3cdbc4b1e9219bb6c564a3c57f399270ea524a23241a43
+size 473848

model1.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e6c28036e8feb26266f749e798cf3262f647b4a96ff351e37a0e3cd171970cf
+size 3588749523

model1.vec

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd0a30892361f8581c04bb32bbfb4e212a1384631be70585d0a64bac3c4d9028
+size 3916574498