--- language: - pt - en library_name: adapter-transformers datasets: - dominguesm/Canarim-Instruct-PTBR-Dataset pipeline_tag: text-generation thumbnail: https://blog.cobasi.com.br/wp-content/uploads/2022/08/AdobeStock_461738919.webp model-index: - name: Caramelinho results: - task: type: text-generation name: Text Generation dataset: name: ENEM Challenge (No Images) type: eduagarcia/enem_challenge split: train args: num_few_shot: 3 metrics: - type: acc value: 21.48 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelinho name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: BLUEX (No Images) type: eduagarcia-temp/BLUEX_without_images split: train args: num_few_shot: 3 metrics: - type: acc value: 22.11 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelinho name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: OAB Exams type: eduagarcia/oab_exams split: train args: num_few_shot: 3 metrics: - type: acc value: 25.15 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelinho name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Assin2 RTE type: assin2 split: test args: num_few_shot: 15 metrics: - type: f1_macro value: 48.97 name: f1-macro - type: pearson value: 19.38 name: pearson source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelinho name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: FaQuAD NLI type: ruanchaves/faquad-nli split: test args: num_few_shot: 15 metrics: - type: f1_macro value: 43.92 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelinho name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HateBR Binary type: eduagarcia/portuguese_benchmark split: test args: num_few_shot: 25 metrics: - type: f1_macro value: 33.97 name: f1-macro - type: f1_macro value: 46.57 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelinho name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: tweetSentBR type: eduagarcia-temp/tweetsentbr split: test args: num_few_shot: 25 metrics: - type: f1_macro value: 56.31 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelinho name: Open Portuguese LLM Leaderboard ---
Caramelo
# Caramelinho ## Adapter Description This adapter was created with the [PEFT](https://github.com/huggingface/peft) library and allowed the base model **Falcon-7b** to be fine-tuned on the [Canarim](https://huggingface.co/datasets/dominguesm/Canarim-Instruct-PTBR-Dataset) by using the method **QLoRA**. ## Model description [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b) ## Intended uses & limitations TBA ## Training and evaluation data TBA ### Training results ### How to use ```py import torch from peft import PeftModel, PeftConfig from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, GenerationConfig peft_model_id = "Bruno/Caramelinho" config = PeftConfig.from_pretrained(peft_model_id) bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, ) tokenizer = AutoTokenizer.from_pretrained(peft_model_id) model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, quantization_config=bnb_config, trust_remote_code=True, device_map={"": 0}) prompt_input = "Abaixo está uma declaração que descreve uma tarefa, juntamente com uma entrada que fornece mais contexto. Escreva uma resposta que conclua corretamente a solicitação.\n\n ### Instrução:\n{instruction}\n\n### Entrada:\n{input}\n\n### Resposta:\n" prompt_no_input = "Abaixo está uma instrução que descreve uma tarefa. Escreva uma resposta que conclua corretamente a solicitação.\n\n### Instrução:\n{instruction}\n\n### Resposta:\n" def create_prompt(instruction, input=None): if input: return prompt_input.format(instruction=instruction, input=input) else: return prompt_no_input.format(instruction=instruction) def generate( instruction, input=None, max_new_tokens=128, temperature=0.1, top_p=0.75, top_k=40, num_beams=4, repetition_penalty=1.7, max_length=512 ): prompt = create_prompt(instruction, input) inputs = tokenizer.encode_plus(prompt, return_tensors="pt", truncation=True, max_length=max_length, padding="longest") input_ids = inputs["input_ids"].to("cuda") attention_mask = inputs["attention_mask"].to("cuda") generation_output = model.generate( input_ids=input_ids, attention_mask=attention_mask, max_length=max_length, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, temperature=temperature, top_p=top_p, top_k=top_k, num_beams=num_beams, repetition_penalty=repetition_penalty, length_penalty=0.8, early_stopping=True, output_scores=True, return_dict_in_generate=True ) output = tokenizer.decode(generation_output.sequences[0], skip_special_tokens=True) return output.split("### Resposta:")[1] instruction = "Descrever como funcionam os computadores quânticos." print("Instrução:", instruction) print("Resposta:", generate(instruction)) ### Saída Instrução: Descrever como funcionam os computadores quânticos. Resposta: Os computadores quânticos são um tipo de computador cuja arquitetura é baseada na mecânica quântica. Os computadores quânticos são capazes de realizar operações matemáticas complexas em um curto espaço de tempo. ### Framework versions - Transformers 4.30.0.dev0 - Pytorch 2.0.1+cu118 - Datasets 2.12.0 - Tokenizers 0.13.3 # [Open Portuguese LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/Bruno/Caramelinho) | Metric | Value | |--------------------------|---------| |Average |**35.32**| |ENEM Challenge (No Images)| 21.48| |BLUEX (No Images) | 22.11| |OAB Exams | 25.15| |Assin2 RTE | 48.97| |Assin2 STS | 19.38| |FaQuAD NLI | 43.92| |HateBR Binary | 33.97| |PT Hate Speech Binary | 46.57| |tweetSentBR | 56.31|