Create README.md

README.md

@@ -0,0 +1,49 @@
+# DNA FM 7B
+  DNA FM 7B is DNA foundation model trained on 10.6 billion nucleotides from 796 species, enabling genome mining, in silico mutagenesis studies, gene expression prediction, and directed sequence generation.
+
+  By scaling model depth while maintaining a short context length of 4000 nucleotides, DNA FM shows substantial improvements across a breadth of tasks in functional genomics using transfer learning, sequence generation, and unsupervised annotation of functional elements. Notably, DNA FM outperforms prior encoder-only architectures without new data, suggesting that new scaling laws are needed to achieve compute-optimal DNA language models.
+
+## Model Architectural Details
+  DNA FM 7B is based on the bidirectional transformer encoder (BERT) architecture with single-nucleotide tokenization, and is optimized using a masked language modeling (MLM) training objective.
+
+  To learn semantically meaningful representations, we employed an BERT-style encoder-only dense transformer architecture. We make minor updates to this architecture to align with current best practices, including using SwiGLU and LayerNorms. Additionally, we use Rotary Positional Embeddings (RoPE), given that DNA syntax does not function based on absolute nucleotide positions but nucleotides interact in highly local and context-specific ways. DNA FM 7B used 32 transformer blocks with embedding size 4,352, where each block contains 32 self-attention heads and feed forward size of 11,584.
+
+  Below are more detailes about the model architecture
+  | Model Arch Component        | Value          |
+| ------------- |:-------------:|
+| Num Attention Heads      | 32  |
+| Num Hidden Layers      | 32       |
+| Hidden Size | 4352       |
+| Intermediate Size | 11584       |
+| Vocab Size | 16      |
+| Context Length | 4000      |
+
+## Pre-training of DNA FM 7B
+Here we briefly introduce the details of pre-training of DNA FM 7B. For more detailed information, please refer to [our paper](https://openreview.net/forum?id=Kis8tVUeNi&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2024%2FWorkshop%2FAIDrugX%2FAuthors%23your-submissions)).
+
+### Data
+To test whether representation capacity has limited the development of DNA language models in previous studies, we utilize the data set and splits from the Nucleotide Transformer. Starting from a total of 812 genomes with 712 for training, 50 for validation, and 50 for testing, we removed 17 entries which had been deleted from NCBI since the original dataset’s publication on Hugging Face. One of these was the important model organism Rattus norvegigus, which we replaced with the current reference genome. This resulted in 696 genomes for training, 50 for validation, and 50 for testing. We pre-trained DNA FM 7B With a total of 10.6 billion training tokens.
+
+### Training Details
+  The weights of our seven billion parameter model occupy over 200GB of memory in 32 bit precision. To train a model of this size, we use model parallelism to split training across 256 H100 GPUs using the Megatron-LM framework. We also employed bfloat16 mixed precision training and FlashAttention-2 to allow for training with large context length at scale. With these optimizations, our maximum possible global batch size was 1024 with a per-device micro batch size of 2. With this configuration, DNA FM 7B took 8 days to train to 100,000 iters. 
+
+## Evaluation of DNA FM 7B
+We evaluate the benefits of pretraining DNA FM 7B by conducting a comprehensive series of experiments related to functional genomics, genome mining, metabolic engineering, synthetic biology, and therapeutics design, covering supervised, unsupervised, and generative objectives. Unless otherwise stated, hyperparameters were determined by optimizing model performance on a 10% validation split of the training data, and models were tested using the checkpoint with the lowest validation loss. For more detailed information, please refer to [our paper](https://openreview.net/forum?id=Kis8tVUeNi&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2024%2FWorkshop%2FAIDrugX%2FAuthors%23your-submissions)).
+
+## Evaluation
+  TODO (@Caleb), we will need to see what results we want to put here.
+
+## How to Use
+  TODO(@Dian Li)
+
+
+## Citation
+Please cite DNA FM using the following BibTeX code:
+```
+@inproceedings{ellington2024accurate,
+title={Accurate and General {DNA} Representations Emerge from Genome Foundation Models at Scale},
+author={Caleb Ellington, Ning Sun, Nicholas Ho, Tianhua Tao, Sazan Mahbub, Yonghao Zhuang, Hongyi Wang, Eric P. Xing, Le Song},
+booktitle={NeurIPS 2024 Workshop on AI for New Drug Modalities},
+year={2024}
+}
+```