README.md

---
license: apache-2.0
library_name: transformers
language:
- en
- pt
pipeline_tag: translation
---
# Transformer-eng-por

## Model Overview

The transformer-eng-por model is a transformer trained for text classification.

### Details

- **Size:** 23,805,216 parameters
- **Model type:** Transformer
- **Optimizer**: `rmsprop` 
- **Number of Epochs:** 30
- **Embbedding dimensionality:** 256
- **Dense dimensionality:** 2048
- **Attention heads:** 8
- **Vocabulary size:** 20000
- **Sequence lenght:** 20
- **Hardware:** Tesla V4
- **Emissions:** Not measured
- **Total Energy Consumption:** Not measured

### How to Use

```python
import tensorflow as tf
import numpy as np
import string
import keras
import re

strip_chars = string.punctuation
strip_chars = strip_chars.replace("[", "")
strip_chars = strip_chars.replace("]", "")


def custom_standardization(input_string):
    lowercase = tf.strings.lower(input_string)
    return tf.strings.regex_replace(lowercase, f"[{re.escape(strip_chars)}]", "")

portuguese_vocabulary_path = hf_hub_download(
    repo_id="AiresPucrs/transformer-eng-por",
    filename="keras_transformer_blocks.py",
    repo_type='model',
    local_dir="./")

from keras_transformer_blocks import TransformerEncoder, PositionalEmbedding, TransformerDecoder

transformer = keras.models.load_model("/content/transformer-eng-por/transformer-eng-por.h5",
    custom_objects={"TransformerEncoder": TransformerEncoder,
        "PositionalEmbedding": PositionalEmbedding,
        "TransformerDecoder": TransformerDecoder})

with open('portuguese_vocabulary.txt', encoding='utf-8', errors='backslashreplace') as fp:
    portuguese_vocab = [line.strip() for line in fp]
    fp.close()

with open('english_vocabulary.txt', encoding='utf-8', errors='backslashreplace') as fp:
    english_vocab = [line.strip() for line in fp]
    fp.close()


target_vectorization = tf.keras.layers.TextVectorization(max_tokens=20000,
                                        output_mode="int",
                                        output_sequence_length=21,
                                        standardize=custom_standardization,
                                        vocabulary=portuguese_vocab)

source_vectorization = tf.keras.layers.TextVectorization(max_tokens=20000,
                                        output_mode="int",
                                        output_sequence_length=20,
                                        vocabulary=english_vocab)

portuguese_index_lookup = dict(zip(range(len(portuguese_vocab)), portuguese_vocab))
max_decoded_sentence_length = 20


def decode_sequence(input_sentence):
    tokenized_input_sentence = source_vectorization([input_sentence])
    decoded_sentence = "[start]"

    for i in range(max_decoded_sentence_length):
        tokenized_target_sentence = target_vectorization([decoded_sentence])[:, :-1]
        predictions = transformer([tokenized_input_sentence, tokenized_target_sentence])
        sampled_token_index = np.argmax(predictions[0, i, :])
        sampled_token = portuguese_index_lookup[sampled_token_index]
        decoded_sentence += " " + sampled_token
        if sampled_token == "[end]":
            break
    return decoded_sentence


eng_sentences =["What is its name?",
                "How old are you?",
                "I know you know where Mary is.",
                "We will show Tom.",
                "What do you all do?",
                "Don't do it!"]

for sentence in eng_sentences:
    print(f"English sentence:\n{sentence}")
    print(f'Portuguese translation:\n{decode_sequence(sentence)}')
    print('-' * 50)
```
This will output the following:

```
English sentence:
What is its name?
Portuguese translation:
[start] qual é o nome dele [end]
--------------------------------------------------
English sentence:
How old are you?
Portuguese translation:
[start] quantos anos você tem [end]
--------------------------------------------------
English sentence:
I know you know where Mary is.
Portuguese translation:
[start] eu sei que você sabe onde mary está [end]
--------------------------------------------------
English sentence:
We will show Tom.
Portuguese translation:
[start] vamos ligar para o tom [end]
--------------------------------------------------
English sentence:
What do you all do?
Portuguese translation:
[start] o que vocês todos nós têm feito [end]
--------------------------------------------------
English sentence:
Don't do it!
Portuguese translation:
[start] não faça isso [end]
--------------------------------------------------
```
## Intended Use

This model was created for research purposes only. Specifically, it was designed to translate sentences from English to Portuguese. We do not recommend any application of this model outside this scope.


## Performance Metrics

Accuracy: 76,46%


## Training Data

 [English-portuguese translation](https://www.kaggle.com/datasets/nageshsingh/englishportuguese-translation).

 The dataset consists of a set of English and Portuguese sentences.

## Limitations

 In  `What is its name?` to `[start] o que é o seu nome [end]`, the `transformer` model makes a gender assumption,
 even though the source sentence wasn't gendered (`[start] qual é o nome dele [end]`).
 Errors like these are common in NLP, algorithmic bias being one of the great problems associated with using language models
 in real applications. In conclusion, we do not recommend using this model in real-world applications.
 It was solely developed for academic and educational purposes.

# Cite as
```latex
@misc{teenytinycastle,
    doi = {10.5281/zenodo.7112065},
    url = {https://github.com/Nkluge-correa/teeny-tiny_castle},
    author = {Nicholas Kluge Corr{\^e}a},
    title = {Teeny-Tiny Castle},
    year = {2024},
    publisher = {GitHub},
    journal = {GitHub repository},
}
```
## License
The transformer-eng-por is licensed under the Apache License, Version 2.0. See the LICENSE file for more details.