| <p align="center" width="100%"> | |
| <img src="assets/logo.png" alt="Vigogne" style="width: 30%; min-width: 300px; display: block; margin: auto;"> | |
| </p> | |
| # LLaMandement-13B: A Llama-2-based French Models for Summarization of French Legislative Proposals. | |
| LLaMandement-13B is a French chat LLM, based on [LLaMA-2-13B](https://ai.meta.com/llama), optimized to summarize of French Legislative Proposals. | |
| ## Model Details | |
| - **Developed by:** [DGFIP](https://www.impots.gouv.fr/presentation-de-la-dgfip-overview-dgfip) : | |
| - yannis.tannier@dgfip.finances.gouv.fr | |
| - joseph.gesnouin@dgfip.finances.gouv.fr | |
| - **Model type:** An auto-regressive language model based on the transformer architecture | |
| - **License:** Llama 2 Community License Agreement | |
| - **Finetuned from model:** [Llama 2](https://arxiv.org/abs/2307.09288) | |
| - **Repository:** https://gitlab.adullact.net/dgfip/projets-ia/llamandement | |
| - **Paper:** working | |
| ## Prompt Template | |
| The prompt for LLaMandement-13B is based on alpaca template : | |
| ``` | |
| Below is an instruction that describes a task. Write a response that appropriately completes the request. | |
| ### Instruction: | |
| {instruction} | |
| ### Response: | |
| ``` | |
| ## How to Get Started with the Model | |
| - Command line interface: https://github.com/lm-sys/FastChat | |
| - APIs (OpenAI API, Huggingface API): https://github.com/lm-sys/FastChat/tree/main#api | |
| ## Training Details | |
| Llamandement-13B is fine-tuned from Llama 2 using Low-Rank Adaptation (LORA). This method is efficient and adds minimal computational load. It introduces additional low-rank parameters, enabling the model to better handle complex legislative language without major changes to the original structure. | |
| **LORA Settings Adjustments:** | |
| - **Learning Rate (LR):** Set to a low value of 2e-5 to ensure stable and gradual improvements. | |
| - **Adaptation Depth (lora_r):** Set at 64, influencing the dimension of the low-rank matrix in LORA. This affected about 0.40% of the model's weights. | |
| - **Decay Rate:** Employed at 0.01 to prevent overfitting to specific legislative text structures. | |
| - **LORA Alpha (α):** Set at 16, it fine-tunes the model's response to legislative text. | |
| - **LORA Dropout:** A rate of 0.1 applied to LORA layers to prevent overfitting and enhance generalization. | |
| - **Optimizer and Scheduler:** Utilized a cosine learning rate scheduler with a warmup ratio of 0.03 for optimal training. | |
| For more information, visit [dgfip.finance.com](http://dgfip.finance.com). Additional details about the training dataset composition can be found [here](http://dgfip.finance.com/training-dataset-info). | |
| ## Citation | |
| Please cite the repo if you use the data, method or code in this repo. | |
| [...] | |