Added Model card for the pose model using details from the paper. (#2)

- Added Model card for the pose model using details from the paper. (234b11885421fe6be0aa3756d1354b5cb930d82d)


Co-authored-by: Girish Salunke <gsalunke@users.noreply.huggingface.co>

README.md

@@ -5,195 +5,265 @@ tags: []
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+Finetuned Vision Transformer (ViT-16) model for classifying the pose of figures in MixTec Codices.
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+This model is designed for classifying the pose of figures depicted in the Mixtec codices(standing/not standing).The codices depict historical and mythological scenes using structured pictorial representations. The models Vision Transformer (ViT-16), was finetuned on a custom-labeled dataset of 1,300 figures extracted from three historical Mixtec codices. 
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
 
-- **Developed by:** [More Information Needed]
+- **Developed by:** [ufdatastudio.com](https://ufdatastudio.com/)
 - **Funded by [optional]:** [More Information Needed]
 - **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
+- **Model type:** Image Classification
+- **Language:** Python
 - **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
+- **Finetuned from model [optional]:** Vision Transformer (ViT-16)
 
-<!-- Provide the basic links for the model. -->
+### Model Sources
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/ufdatastudio/mixteclabeling
+- **Paper:** [Analyzing Finetuned Vision Models for Mixtec Codex Interpretation](https://ufdatastudio.com/papers/webber2024analyzing.pdf)
+- **Poster** https://ufdatastudio.com/papers/webber2024analyzing-poster.pdf
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+This model is intended for the classification of figures in historical Mixtec codices. The classification of pose assists in the interpretation of ancient Mixtec manuscripts, contributing to historical and anthropological research. 
 
-[More Information Needed]
+### Downstream Use
 
-### Downstream Use [optional]
+This model may be used for more advanced tasks such as relationship extraction between figures within a codex scene, potentially helping to reconstruct the narratives depicted in the codices. 
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Out-of-Scope Use
 
-[More Information Needed]
+Using the model for classification on datasets unrelated to Mixtec codices or datasets not following similar pictographic systems could yield inaccurate results. The model may not generalize well to modern or non-Mesoamerican artistic depictions. 
 
-### Out-of-Scope Use
+## Bias, Risks, and Limitations
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
++ The model has adopted the use of  pretrained classifiers, each trained on data not specific to our domain. 
 
-[More Information Needed]
++ The models inherit all biases previously encoded in the model. We have not investigated how these biases may affect downstream tasks.
 
-## Bias, Risks, and Limitations
++ The finetuned models generated few errors in our investigation, however, we are unaware of how these biases may result in unintended effects.
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
++ This work is an initial investigation into Mixtec and low- resource, semasiographic languages. We are prohibited from deeper explorations until we align our research direction with present communal, cultural, and anthropological needs. Support from Mixtec domain experts and native Mixtec speakers is essential for continued development.
 
-[More Information Needed]
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+Given that the model can reliably classify figures from a low-resource dataset, this research opens the door for further processing and analysis of Mixtec Codices. The codices themselves are highly structured and carry a narrative woven through each scene. Finetuned state-of-the-art models could be combined to classify segmented figures within a scene, as well as classify the relationship between figures. These relationships would then be used to extract the narrative from a codex, as defined by domain experts.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+```python 
+
+from transformers import ViTFeatureExtractor,ViTForImageClassification 
+from PIL import Image 
+import torch 
+import requests 
+from io import BytesIO 
+ 
+# Load the feature extractor and model 
+model_name = "ufdatastudio/vit-pose" 
+feature_extractor = ViTFeatureExtractor.from_pretrained(model_name) 
+model = ViTForImageClassification.from_pretrained(model_name) 
+ 
+img = Image.open("<link_to_image>").convert("RGB") 
+ 
+# Preprocess the image 
+inputs = feature_extractor(images=img, return_tensors="pt") 
+ 
+# Run inference (classify the image) 
+with torch.no_grad(): 
+    outputs = model(**inputs) 
+ 
+# Get predicted class 
+predicted_class_idx = outputs.logits.argmax(-1).item() 
+labels = model.config.id2label  # get labels 
+predicted_label = labels[predicted_class_idx] 
+ 
+# Print the result 
+print(f"Predicted Label: {predicted_label}") 
+``` 
 
 ## Training Details
 
 ### Training Data
+The dataset used for the training of this model can be found at: https://huggingface.co/datasets/ufdatastudio/mixtec-figures
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+#### **Dataset Generation**
 
-[More Information Needed]
++ Extracted labelled data from 3 Codices:
 
-### Training Procedure
+    1. **Vindobonensis Mexicanus (65 pages)**: Describes both the mythological and historical founding of the first Mixtec kingdoms.
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+    2. **Selden (20 pages)**: Follows the founding of the kingdom of Jaltepec and its ruler, Lady 6 Monkey.
 
-#### Preprocessing [optional]
+    3. **Zouche-Nuttall (facsimile edition (40 pages))**: Illustrates the life and conquests of Lord 8 Deer Jaguar Claw, but also details the histories of his ancestors.
 
-[More Information Needed]
+> Note:  Other Mixtex Codices are extant, but their condition is degraded and not amenable to our current machine-learning pipeline. Each codex is made of deerskin folios, and each folio comprises two pages.
 
++ **Extraction Method**: We used the [Segment Anything Model (SAM) from Facebook AI Research](https://segment-anything.com/) to extract individual figures from the three source codices.
 
-#### Training Hyperparameters
+    + Each figure was annotated according to the page it was found, its quality as either a, b, or c, and its order within the page.
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+        **a**. quality rating indicated the entire figure was intact, regard- less of minor blemishes or cracking, and could be classified by a human annotator as man or woman, standing or not.
 
-#### Speeds, Sizes, Times [optional]
+        **b**. rating means that while the previous characteristics of the figure could be de- termined, significant portions of the figures were missing or damaged.
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+        **c**. rated figures were missing most of the definable characteristics humans could use to classify the sample.
 
-[More Information Needed]
++ **Data Labelling**: After figure segmentation and grading, we added classification labels to each figure (standing/not standing).
 
-## Evaluation
+    + **Literature used for evaluation of figures**: Boone 2000; Smith, 1973; Jansen, 1988; Williams, 2013; Lopez, 2021.
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+    + **Criteria used to determine standing and not standing**: If the figure is clearly on two feet and in an upright position, it is labeled standing, and any other position is labeled not standing.
 
-### Testing Data, Factors & Metrics
+    + Two team members tagged the images for both categories independently and then verified the results with each other using the process of inter-rater reliability.
 
-#### Testing Data
+### Training Procedure
 
-<!-- This should link to a Dataset Card if possible. -->
+#### **Preprocessing**
 
-[More Information Needed]
++ Figures are moved to tensors and then normalized to 224x224 pixels.
 
-#### Factors
++ Loss function is biased by weighting each class in the loss function by its inverse.
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
++ Due to the overall limited number of figures, and to prevent overfitting, the entire dataset was augmented by using random flips and blocking to increase the number of samples for training.
 
-[More Information Needed]
++ The dataset is split into training, testing, and validation sets, 60%, 20%, and 20% respectively.
 
-#### Metrics
++ Eight reference images were set aside to monitor which features of pose are prevalent in activation and attention maps throughout training.
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+#### **Model Training**
 
-[More Information Needed]
++  We fine-tuned popular vision model ViT-16 to perform classification tasks and improve computational efficiency.
 
-### Results
++ Imported the model and its pre-trained weights from the PyTorch library, then unfroze the last four layers and heads of the model for training, as they are responsible for learning complex features specific to our classification tasks.
 
-[More Information Needed]
++ Replaced the fully connected layer by one matching our binary classification task. 
 
-#### Summary
++ Before the first and after the last epoch of training, an a an attention map is output for each reference image. 
 
 
+### **Hyperparamter Tuning**
 
-## Model Examination [optional]
++ Experimented with different batch sizes, ranging from 32 to 128, and opted for an average value of 64 as no size significantly outperformed the others.
 
-<!-- Relevant interpretability work for the model goes here -->
++ Selected the loss function and optimizer according to the best practices associated with ViT.
 
-[More Information Needed]
++ Hyperparameter investigations revealed that the accuracy for training and validation converged around 100 epochs and the ideal learning rate was 0.00025.
 
-## Environmental Impact
+### **Model Evaluation**
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
++ For each training and validation run, we collected metrics such as accuracy, F1, recall, loss, and precision. 
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
++ The testing accuracy was around 98% with a standard deviation of 1%.
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
+### Testing Data, Factors & Metrics
 
-### Model Architecture and Objective
+#### Testing Data
 
-[More Information Needed]
+The test set was 20% of the overall dataset, comprising 260 figures from all three codices. 
 
-### Compute Infrastructure
 
-[More Information Needed]
+#### Factors
+
+In the dataset the number of images labelled as 'Not standing' outweighs images labelled as 'standing'. The reason for this is unclear, although given the number of ceremonies that each codex describes, which entails a seated or kneeling position, this balance intuitively makes sense.
 
-#### Hardware
+#### Metrics
 
-[More Information Needed]
+The model’s performance was evaluated using accuracy, precision, recall, and F1 scores. Both models performed with around 98% accuracy, with ViT-16 outperforming VGG-16 in some configurations. 
 
-#### Software
+### Results
 
-[More Information Needed]
+The purpose of bulding the model was to answer the questions:
+ 
+   1. **Can transformer-based models be finetuned to classify figures from a Mixtec Codices dataset?**  
 
-## Citation [optional]
+   Yes!, the model achieved great results across training, validation, and testing phases when using an appropriate learning rate.
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+   2. **Does the model identify the same features experts do?**
 
-**BibTeX:**
+   + We assigned reference images for each class (man and woman, and standing/not standing) to understand which features each model learned, as well as to compare these learned features to those highlighted by experts.
 
-[More Information Needed]
+   + During training, we generated visualizations of activation and attention per pixel to view how the models learned important features over time.
 
-**APA:**
+   <!-- ![Alt text](image-1.png) -->
 
-[More Information Needed]
+   + The ViT model assigned higher attention to areas corresponding to loincloths on man and showed increased attention to the poncho area on a woman.
 
-## Glossary [optional]
+   + To verify that the model is indeed identifying the same features noted in literature, we masked attributes on the reference images.
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+   + We extended our reference image set by adding three variations to each image: either blocked hair, blocked skirt, or both for woman. This process was replicated for the two features indicative of man. 
 
-[More Information Needed]
+   + ViT correctly predicted 100% of the unblocked reference images, 79% of the singly blocked images, and 63% of the double blocked images.
 
-## More Information [optional]
+   + For the doubly blocker images the model fails to find defined areas of attention. This verifies that the model is learning features defined in literature.
 
-[More Information Needed]
 
-## Model Card Authors [optional]
+#### Summary
 
-[More Information Needed]
+ We presented a low-resource dataset of figures from three Mixtec codices: Zouche-Nuttall, Selden, and Vindobonensis Mexicanus I. We extracted the figures using Segment Anything Model and labeled them according to pose, a critical feature used to understand Mixtec codices. Using this novel dataset, we finetuned the last few layers transformer-based foundational models ViT-16, to classify figures as standing or not standing. We confirmed that the model is learning the features said to be relevant by experts using class activation maps and targeted blocking of said features. 
 
-## Model Card Contact
+## Environmental Impact
 
-[More Information Needed]
+We have not yet explored more environmentally efficient models. The environmental impact is the same as that of the Vision Transformer models.
+
+## Technical Specifications
+
+### Compute Infrastructure
+
+#### Hardware
+
+Model training and inference were performed on an Nvidia A100 on the HiPerGator cluster using PyTorch 2.1 and CUDA 11.
+
+#### Software
+
+PyTorch framework
+
+## Citation
+
+**BibTeX:**
+```BibTeX
+@inproceedings{webber-etal-2024-analyzing,
+    title = "Analyzing Finetuned Vision Models for {M}ixtec Codex Interpretation",
+    author = "Webber, Alexander  and
+      Sayers, Zachary  and
+      Wu, Amy  and
+      Thorner, Elizabeth  and
+      Witter, Justin  and
+      Ayoubi, Gabriel  and
+      Grant, Christan",
+    editor = "Mager, Manuel  and
+      Ebrahimi, Abteen  and
+      Rijhwani, Shruti  and
+      Oncevay, Arturo  and
+      Chiruzzo, Luis  and
+      Pugh, Robert  and
+      von der Wense, Katharina",
+    booktitle = "Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024)",
+    month = jun,
+    year = "2024",
+    address = "Mexico City, Mexico",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2024.americasnlp-1.6",
+    doi = "10.18653/v1/2024.americasnlp-1.6",
+    pages = "42--49",
+    abstract = "Throughout history, pictorial record-keeping has been used to document events, stories, and concepts. A popular example of this is the Tzolk{'}in Maya Calendar. The pre-Columbian Mixtec society also recorded many works through graphical media called codices that depict both stories and real events. Mixtec codices are unique because the depicted scenes are highly structured within and across documents. As a first effort toward translation, we created two binary classification tasks over Mixtec codices, namely, gender and pose. The composition of figures within a codex is essential for understanding the codex{'}s narrative. We labeled a dataset with around 1300 figures drawn from three codices of varying qualities. We finetuned the Visual Geometry Group 16 (VGG-16) and Vision Transformer 16 (ViT-16) models, measured their performance, and compared learned features with expert opinions found in literature. The results show that when finetuned, both VGG and ViT perform well, with the transformer-based architecture (ViT) outperforming the CNN-based architecture (VGG) at higher learning rates. We are releasing this work to allow collaboration with the Mixtec community and domain scientists.",
+}
+```
+
+## Glossary
+
+Figures: Representations of people or gods in Mixtec mythology and are composed of different outfits, tools, and positions. Their names are represented by icons placed near their position on a page.
+
+<!-- ## Model Card Authors [optional] -->
+
+## Model Card Contact
+https://ufdatastudio.com/contact/