Update README.md

README.md

@@ -1,86 +1,103 @@
----
-language: en
-license: apache-2.0
-datasets:
-- nyu-mll/glue
----
-
-# LoNAS Model Card: lonas-bert-base-glue
-
-The super-networks fine-tuned on BERT-base with [GLUE benchmark](https://gluebenchmark.com/) using LoNAS.
-
-## Model Details
-
-### Information
-
-- **Model name:** lonas-bert-base-glue
-- **Base model:** [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)
-- **Subnetwork version:** Super-network
-- **NNCF Configurations:** [nncf_config/glue](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS/nncf_config/glue)
-
-### Adapter Configuration
-
-- **LoRA rank:** 8
-- **LoRA alpha:** 16
-- **LoRA target modules:** query, value
-
-
-### Training and Evaluation
-
-[GLUE benchmark](https://gluebenchmark.com/)
-
-### Training Hyperparameters
-
-| Task       | RTE  | MRPC | STS-B | CoLA | SST-2 | QNLI | QQP  | MNLI |
-|------------|------|------|-------|------|-------|------|------|------|
-| Epoch      | 80   | 35   | 60    | 80   | 60    | 80   | 60   | 40   |
-| Batch size | 32   | 32   | 64    | 64   | 64    | 64   | 64   | 64   |
-| Learning rate | 3e-4 | 5e-4 | 5e-4  | 3e-4 | 3e-4  | 4e-4 | 3e-4 | 4e-4 |
-| Max length | 128  | 128  | 128   | 128  | 128   | 256  | 128  | 128  |
-
-## How to use
-
-Refer to [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS/running_commands](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS/running_commands):
-```bash
-CUDA_VISIBLE_DEVICES=${DEVICES} python run_glue.py \
-    --task_name ${TASK} \
-    --model_name_or_path bert-base-uncased \
-    --do_eval \
-    --do_search \
-    --per_device_eval_batch_size 64 \
-    --max_seq_length ${MAX_LENGTH} \
-    --lora \
-    --lora_weights lonas-bert-base-glue/lonas-bert-base-${TASK} \
-    --nncf_config nncf_config/glue/nncf_lonas_bert_base_${TASK}.json \
-    --output_dir lonas-bert-base-glue/lonas-bert-base-${TASK}/results
-```
-
-## Evaluation Results
-
-Results of the optimal sub-network discoverd from the super-network:
-
-| Method      | Trainable Parameter Ratio | GFLOPs     | RTE   | MRPC  | STS-B | CoLA  | SST-2 | QNLI  | QQP   | MNLI  | AVG       |
-|-------------|---------------------------|------------|-------|-------|-------|-------|-------|-------|-------|-------|-----------|
-| LoRA        | 0.27%                     | 11.2       | 65.85 | 84.46 | 88.73 | 57.58 | 92.06 | 90.62 | 89.41 | 83.00 | 81.46     |
-| **LoNAS**   | 0.27%                     | **8.0**    | 70.76 | 88.97 | 88.28 | 61.12 | 93.23 | 91.21 | 88.55 | 82.00 | **83.02** |
-
-
-## Model Sources
-
-- **Repository:** [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS)
-- **Paper:** [LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models]()
-
-## Citation
-
-```bibtex
-@article{munoz2024lonas,
-  title = {LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models},
-  author={J. Pablo Munoz and Jinjie Yuan and Yi Zheng and Nilesh Jain},
-  journal={},
-  year={2024}
-}
-```
-
-## License
-
-Apache-2.0
+---
+language: en
+license: apache-2.0
+datasets:
+- nyu-mll/glue
+---
+
+# LoNAS Model Card: lonas-bert-base-glue
+
+The super-networks fine-tuned on BERT-base with [GLUE benchmark](https://gluebenchmark.com/) using LoNAS.
+
+## Model Details
+
+### Information
+
+- **Model name:** lonas-bert-base-glue
+- **Base model:** [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)
+- **Subnetwork version:** Super-network
+- **NNCF Configurations:** [nncf_config/glue](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS/nncf_config/glue)
+
+### Adapter Configuration
+
+- **LoRA rank:** 8
+- **LoRA alpha:** 16
+- **LoRA target modules:** query, value
+
+
+### Training and Evaluation
+
+[GLUE benchmark](https://gluebenchmark.com/)
+
+### Training Hyperparameters
+
+| Task       | RTE  | MRPC | STS-B | CoLA | SST-2 | QNLI | QQP  | MNLI |
+|------------|------|------|-------|------|-------|------|------|------|
+| Epoch      | 80   | 35   | 60    | 80   | 60    | 80   | 60   | 40   |
+| Batch size | 32   | 32   | 64    | 64   | 64    | 64   | 64   | 64   |
+| Learning rate | 3e-4 | 5e-4 | 5e-4  | 3e-4 | 3e-4  | 4e-4 | 3e-4 | 4e-4 |
+| Max length | 128  | 128  | 128   | 128  | 128   | 256  | 128  | 128  |
+
+## How to use
+
+Refer to [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS/running_commands](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS/running_commands):
+```bash
+CUDA_VISIBLE_DEVICES=${DEVICES} python run_glue.py \
+    --task_name ${TASK} \
+    --model_name_or_path bert-base-uncased \
+    --do_eval \
+    --do_search \
+    --per_device_eval_batch_size 64 \
+    --max_seq_length ${MAX_LENGTH} \
+    --lora \
+    --lora_weights lonas-bert-base-glue/lonas-bert-base-${TASK} \
+    --nncf_config nncf_config/glue/nncf_lonas_bert_base_${TASK}.json \
+    --output_dir lonas-bert-base-glue/lonas-bert-base-${TASK}/results
+```
+
+## Evaluation Results
+
+Results of the optimal sub-network discoverd from the super-network:
+
+| Method      | Trainable Parameter Ratio | GFLOPs     | RTE   | MRPC  | STS-B | CoLA  | SST-2 | QNLI  | QQP   | MNLI  | AVG       |
+|-------------|---------------------------|------------|-------|-------|-------|-------|-------|-------|-------|-------|-----------|
+| LoRA        | 0.27%                     | 11.2       | 65.85 | 84.46 | 88.73 | 57.58 | 92.06 | 90.62 | 89.41 | 83.00 | 81.46     |
+| **LoNAS**   | 0.27%                     | **8.0**    | 70.76 | 88.97 | 88.28 | 61.12 | 93.23 | 91.21 | 88.55 | 82.00 | **83.02** |
+
+
+## Model Sources
+
+**Repository:** [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS)
+
+**Paper:** 
+- [LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models](https://aclanthology.org/2024.lrec-main.940)
+- [Low-Rank Adapters Meet Neural Architecture Search for LLM Compression](https://arxiv.org/abs/2501.16372)
+
+## Citation
+
+```bibtex
+@inproceedings{munoz-etal-2024-lonas,
+    title = "{L}o{NAS}: Elastic Low-Rank Adapters for Efficient Large Language Models",
+    author = "Munoz, Juan Pablo  and
+      Yuan, Jinjie  and
+      Zheng, Yi  and
+      Jain, Nilesh",
+    editor = "Calzolari, Nicoletta  and
+      Kan, Min-Yen  and
+      Hoste, Veronique  and
+      Lenci, Alessandro  and
+      Sakti, Sakriani  and
+      Xue, Nianwen",
+    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
+    month = may,
+    year = "2024",
+    address = "Torino, Italia",
+    publisher = "ELRA and ICCL",
+    url = "https://aclanthology.org/2024.lrec-main.940",
+    pages = "10760--10776",
+}
+```
+
+## License
+
+Apache-2.0