Model Card for tesolnet/tari01

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: TARI
• Model type: GPT-2 variant (distilled version)
• Language(s) (NLP): English
• License: MIT
• Finetuned from model: distilgpt2

Uses

Direct Use

This model can be used for text generation tasks such as generating text based on a prompt and creating chatbots.

Downstream Use [optional]

This model can be further fine-tuned for specific tasks such as sentiment analysis, question answering, or other NLP tasks requiring text generation.

Out-of-Scope Use

The model should not be used for generating harmful, misleading, or malicious content. It may not perform well on tasks requiring understanding of context beyond a few sentences or paragraphs.

Bias, Risks, and Limitations

This model, like all language models, can produce biased or harmful text based on the data it was trained on. Users should be aware of these limitations and use the model with caution.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. More information is needed for further recommendations.

How to Get Started with the Model

To get started with the model, use the transformers library from Hugging Face. Load the model and tokenizer with the following identifiers: tesolnet/tari01.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("tesolnet/tari01")
tokenizer = AutoTokenizer.from_pretrained("tesolnet/tari01")

inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(inputs.input_ids, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The model was fine-tuned on 100 ebooks about computational linguistics, preprocessed and tokenized for training.

Training Procedure

Preprocessing [optional]

The text data was tokenized using the AutoTokenizer from the transformers library with a maximum token length of 128.

Training Hyperparameters

• Training regime: Mixed precision (fp16)
• Learning rate: 2e-5
• Batch size: 2
• Epochs: 1
• Weight decay: 0.01

Speeds, Sizes, Times [optional]

• Training time: Approximately 3.85 hours

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluation was performed on a subset of the training data held out for validation purposes.

Factors

Evaluation factors included token accuracy and perplexity on the validation dataset.

Metrics

Evaluation metrics included perplexity, as it measures the model's ability to predict the next token in a sequence.

Results

[More Information Needed]

Summary

The model achieved satisfactory results for text generation tasks based on the validation metrics.

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA GeForce RTX 4090 (2 GPUs)
• Hours used: 3.85 hours

Technical Specifications [optional]

Model Architecture and Objective

The model is a distilled version of GPT-2, fine-tuned for text generation tasks.

Compute Infrastructure

Hardware

Training was performed on two NVIDIA GeForce RTX 4090 GPUs.

Software

• OS: Ubuntu 22.04
• Libraries: transformers, torch, safetensors