README.md

---
license: gemma
base_model: google/paligemma-3b-pt-224
tags:
- generated_from_trainer
datasets:
- AffectNet
model-index:
- name: paligemma_emotion_
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# FaceScanPaliGemma_Emotion


``` python

from PIL import Image
import torch
from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig, TrainingArguments, Trainer


model = PaliGemmaForConditionalGeneration.from_pretrained('NYUAD-ComNets/FaceScanPaliGemma_Emotion',torch_dtype=torch.bfloat16)

input_text = "what is the emotion of the person in the image?"

processor = PaliGemmaProcessor.from_pretrained("google/paligemma-3b-pt-224")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model.to(device)


input_image = Image.open('image_path')
inputs = processor(text=input_text, images=input_image, padding="longest", do_convert_rgb=True, return_tensors="pt").to(device)
inputs = inputs.to(dtype=model.dtype)
      
with torch.no_grad():
          output = model.generate(**inputs, max_length=500)
result=processor.decode(output[0], skip_special_tokens=True)[len(input_text):].strip()


```


## Model description

This model is a fine-tuned version of [google/paligemma-3b-pt-224](https://huggingface.co/google/paligemma-3b-pt-224) on the AffectNet dataset. 
The model aims to classify the  emotion of face image or image with one person into eight categoris such as 'neutral', 'happy', 'sad', 'surprise', 'fear', 'disgust',
'anger', 'contempt'


## Model Performance
Accuracy: 59.4 %,   F1 score: 59 %


## Intended uses & limitations

This model is used for research purposes

## Training and evaluation data

AffectNet dataset was used for training and validating the model


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 2
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2
- num_epochs: 5

### Training results



### Framework versions

- Transformers 4.42.4
- Pytorch 2.1.2+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1



# BibTeX entry and citation info

```

@article{aldahoul2024exploring,
  title={Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age},
  author={AlDahoul, Nouar and Tan, Myles Joshua Toledo and Kasireddy, Harishwar Reddy and Zaki, Yasir},
  journal={arXiv preprint arXiv:2410.24148},
  year={2024}
}

@misc{ComNets,
      url={https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Emotion](https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Emotion)},
      title={FaceScanPaliGemma_Emotion},
      author={Nouar AlDahoul, Yasir Zaki}
}