library_name: transformers
pipeline_tag: keypoint-detection
Model Card for Model ID
Finetuned Vision Transformer (ViT-16) model for classifying the pose of figures in MixTec Codices.
Model Details
Model Description
This model is designed for classifying the pose of figures depicted in the Mixtec codices(standing/not standing).The codices depict historical and mythological scenes using structured pictorial representations. The models Vision Transformer (ViT-16), was finetuned on a custom-labeled dataset of 1,300 figures extracted from three historical Mixtec codices.
- Developed by: ufdatastudio.com
- Funded by [optional]: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Model type: Image Classification
- Language: Python
- License: [More Information Needed]
- Finetuned from model [optional]: Vision Transformer (ViT-16)
Model Sources
- Repository: https://github.com/ufdatastudio/mixteclabeling
- Paper: Analyzing Finetuned Vision Models for Mixtec Codex Interpretation
- Poster https://ufdatastudio.com/papers/webber2024analyzing-poster.pdf
Uses
Direct Use
This model is intended for the classification of figures in historical Mixtec codices. The classification of pose assists in the interpretation of ancient Mixtec manuscripts, contributing to historical and anthropological research.
Downstream Use
This model may be used for more advanced tasks such as relationship extraction between figures within a codex scene, potentially helping to reconstruct the narratives depicted in the codices.
Out-of-Scope Use
Using the model for classification on datasets unrelated to Mixtec codices or datasets not following similar pictographic systems could yield inaccurate results. The model may not generalize well to modern or non-Mesoamerican artistic depictions.
Bias, Risks, and Limitations
The model has adopted the use of pretrained classifiers, each trained on data not specific to our domain.
The models inherit all biases previously encoded in the model. We have not investigated how these biases may affect downstream tasks.
The finetuned models generated few errors in our investigation, however, we are unaware of how these biases may result in unintended effects.
This work is an initial investigation into Mixtec and low- resource, semasiographic languages. We are prohibited from deeper explorations until we align our research direction with present communal, cultural, and anthropological needs. Support from Mixtec domain experts and native Mixtec speakers is essential for continued development.
Recommendations
Given that the model can reliably classify figures from a low-resource dataset, this research opens the door for further processing and analysis of Mixtec Codices. The codices themselves are highly structured and carry a narrative woven through each scene. Finetuned state-of-the-art models could be combined to classify segmented figures within a scene, as well as classify the relationship between figures. These relationships would then be used to extract the narrative from a codex, as defined by domain experts.
How to Get Started with the Model
from transformers import ViTFeatureExtractor,ViTForImageClassification
from PIL import Image
import torch
import requests
from io import BytesIO
# Load the feature extractor and model
model_name = "ufdatastudio/vit-pose"
feature_extractor = ViTFeatureExtractor.from_pretrained(model_name)
model = ViTForImageClassification.from_pretrained(model_name)
img = Image.open("<link_to_image>").convert("RGB")
# Preprocess the image
inputs = feature_extractor(images=img, return_tensors="pt")
# Run inference (classify the image)
with torch.no_grad():
outputs = model(**inputs)
# Get predicted class
predicted_class_idx = outputs.logits.argmax(-1).item()
labels = model.config.id2label # get labels
predicted_label = labels[predicted_class_idx]
# Print the result
print(f"Predicted Label: {predicted_label}")
Training Details
Training Data
The dataset used for the training of this model can be found at: https://huggingface.co/datasets/ufdatastudio/mixtec-figures
Dataset Generation
Extracted labelled data from 3 Codices:
Vindobonensis Mexicanus (65 pages): Describes both the mythological and historical founding of the first Mixtec kingdoms.
Selden (20 pages): Follows the founding of the kingdom of Jaltepec and its ruler, Lady 6 Monkey.
Zouche-Nuttall (facsimile edition (40 pages)): Illustrates the life and conquests of Lord 8 Deer Jaguar Claw, but also details the histories of his ancestors.
Note: Other Mixtex Codices are extant, but their condition is degraded and not amenable to our current machine-learning pipeline. Each codex is made of deerskin folios, and each folio comprises two pages.
Extraction Method: We used the Segment Anything Model (SAM) from Facebook AI Research to extract individual figures from the three source codices.
Each figure was annotated according to the page it was found, its quality as either a, b, or c, and its order within the page.
a. quality rating indicated the entire figure was intact, regard- less of minor blemishes or cracking, and could be classified by a human annotator as man or woman, standing or not.
b. rating means that while the previous characteristics of the figure could be de- termined, significant portions of the figures were missing or damaged.
c. rated figures were missing most of the definable characteristics humans could use to classify the sample.
Data Labelling: After figure segmentation and grading, we added classification labels to each figure (standing/not standing).
Literature used for evaluation of figures: Boone 2000; Smith, 1973; Jansen, 1988; Williams, 2013; Lopez, 2021.
Criteria used to determine standing and not standing: If the figure is clearly on two feet and in an upright position, it is labeled standing, and any other position is labeled not standing.
Two team members tagged the images for both categories independently and then verified the results with each other using the process of inter-rater reliability.
Training Procedure
Preprocessing
Figures are moved to tensors and then normalized to 224x224 pixels.
Loss function is biased by weighting each class in the loss function by its inverse.
Due to the overall limited number of figures, and to prevent overfitting, the entire dataset was augmented by using random flips and blocking to increase the number of samples for training.
The dataset is split into training, testing, and validation sets, 60%, 20%, and 20% respectively.
Eight reference images were set aside to monitor which features of pose are prevalent in activation and attention maps throughout training.
Model Training
We fine-tuned popular vision model ViT-16 to perform classification tasks and improve computational efficiency.
Imported the model and its pre-trained weights from the PyTorch library, then unfroze the last four layers and heads of the model for training, as they are responsible for learning complex features specific to our classification tasks.
Replaced the fully connected layer by one matching our binary classification task.
Before the first and after the last epoch of training, an a an attention map is output for each reference image.
Hyperparamter Tuning
Experimented with different batch sizes, ranging from 32 to 128, and opted for an average value of 64 as no size significantly outperformed the others.
Selected the loss function and optimizer according to the best practices associated with ViT.
Hyperparameter investigations revealed that the accuracy for training and validation converged around 100 epochs and the ideal learning rate was 0.00025.
Model Evaluation
For each training and validation run, we collected metrics such as accuracy, F1, recall, loss, and precision.
The testing accuracy was around 98% with a standard deviation of 1%.
Testing Data, Factors & Metrics
Testing Data
The test set was 20% of the overall dataset, comprising 260 figures from all three codices.
Factors
In the dataset the number of images labelled as 'Not standing' outweighs images labelled as 'standing'. The reason for this is unclear, although given the number of ceremonies that each codex describes, which entails a seated or kneeling position, this balance intuitively makes sense.
Metrics
The model’s performance was evaluated using accuracy, precision, recall, and F1 scores. Both models performed with around 98% accuracy, with ViT-16 outperforming VGG-16 in some configurations.
Results
The purpose of bulding the model was to answer the questions:
- Can transformer-based models be finetuned to classify figures from a Mixtec Codices dataset?
Yes!, the model achieved great results across training, validation, and testing phases when using an appropriate learning rate.
- Does the model identify the same features experts do?
We assigned reference images for each class (man and woman, and standing/not standing) to understand which features each model learned, as well as to compare these learned features to those highlighted by experts.
During training, we generated visualizations of activation and attention per pixel to view how the models learned important features over time.
The ViT model assigned higher attention to areas corresponding to loincloths on man and showed increased attention to the poncho area on a woman.
To verify that the model is indeed identifying the same features noted in literature, we masked attributes on the reference images.
We extended our reference image set by adding three variations to each image: either blocked hair, blocked skirt, or both for woman. This process was replicated for the two features indicative of man.
ViT correctly predicted 100% of the unblocked reference images, 79% of the singly blocked images, and 63% of the double blocked images.
For the doubly blocker images the model fails to find defined areas of attention. This verifies that the model is learning features defined in literature.
Summary
We presented a low-resource dataset of figures from three Mixtec codices: Zouche-Nuttall, Selden, and Vindobonensis Mexicanus I. We extracted the figures using Segment Anything Model and labeled them according to pose, a critical feature used to understand Mixtec codices. Using this novel dataset, we finetuned the last few layers transformer-based foundational models ViT-16, to classify figures as standing or not standing. We confirmed that the model is learning the features said to be relevant by experts using class activation maps and targeted blocking of said features.
Environmental Impact
We have not yet explored more environmentally efficient models. The environmental impact is the same as that of the Vision Transformer models.
Technical Specifications
Compute Infrastructure
Hardware
Model training and inference were performed on an Nvidia A100 on the HiPerGator cluster using PyTorch 2.1 and CUDA 11.
Software
PyTorch framework
Citation
BibTeX:
@inproceedings{webber-etal-2024-analyzing,
title = "Analyzing Finetuned Vision Models for {M}ixtec Codex Interpretation",
author = "Webber, Alexander and
Sayers, Zachary and
Wu, Amy and
Thorner, Elizabeth and
Witter, Justin and
Ayoubi, Gabriel and
Grant, Christan",
editor = "Mager, Manuel and
Ebrahimi, Abteen and
Rijhwani, Shruti and
Oncevay, Arturo and
Chiruzzo, Luis and
Pugh, Robert and
von der Wense, Katharina",
booktitle = "Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024)",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.americasnlp-1.6",
doi = "10.18653/v1/2024.americasnlp-1.6",
pages = "42--49",
abstract = "Throughout history, pictorial record-keeping has been used to document events, stories, and concepts. A popular example of this is the Tzolk{'}in Maya Calendar. The pre-Columbian Mixtec society also recorded many works through graphical media called codices that depict both stories and real events. Mixtec codices are unique because the depicted scenes are highly structured within and across documents. As a first effort toward translation, we created two binary classification tasks over Mixtec codices, namely, gender and pose. The composition of figures within a codex is essential for understanding the codex{'}s narrative. We labeled a dataset with around 1300 figures drawn from three codices of varying qualities. We finetuned the Visual Geometry Group 16 (VGG-16) and Vision Transformer 16 (ViT-16) models, measured their performance, and compared learned features with expert opinions found in literature. The results show that when finetuned, both VGG and ViT perform well, with the transformer-based architecture (ViT) outperforming the CNN-based architecture (VGG) at higher learning rates. We are releasing this work to allow collaboration with the Mixtec community and domain scientists.",
}
Glossary
Figures: Representations of people or gods in Mixtec mythology and are composed of different outfits, tools, and positions. Their names are represented by icons placed near their position on a page.