Major update. Support for 15 LLMs, World Flora Online taxonomy validation, geolocation, 2 OCR methods, significant UI changes, stability improvements, consistent JSON parsing

README.md

@@ -1,13 +1,343 @@
+# VoucherVision
+
+[![VoucherVision](https://LeafMachine.org/img/VV_Logo.png "VoucherVision")](https://LeafMachine.org/)
+
+Table of Contents
+=================
+
+* [Table of Contents](#table-of-contents)
+* [About](#about)
+* [Roadmap and New Features List](#roadmap-and-new-features-list)
+* [Try our public demo!](#try-our-public-demo)
+* [Installing VoucherVision](#installing-VoucherVision)
+   * [Prerequisites](#prerequisites)
+   * [Installation - Cloning the VoucherVision Repository](#installation---cloning-the-VoucherVision-repository)
+   * [About Python Virtual Environments](#about-python-virtual-environments)
+   * [Installation - Windows 10+](#installation---windows-10)
+      * [Virtual Environment](#virtual-environment-1)
+      * [Installing Packages](#installing-packages-1)
+   * [Troubleshooting CUDA](#troubleshooting-cuda)
+* [Create a Desktop Shortcut to Launch VoucherVision GUI](#create-a-desktop-shortcut-to-launch-vouchervision-gui)
+* [Run VoucherVision](#run-vouchervision)
+    * [Setting up API key](#setting-up-api-key)
+    * [Check GPU](#check-gpu)
+    * [Run Tests](#run-tests)
+    * [Starting VoucherVision](#starting-vouchervision)
+    * [Azure Instances of OpenAI](#azure-instances-of-openai)
+* [Custom Prompt Builder](#custom-prompt-builder)
+    * [Load, Build, Edit](#load-build-edit)
+    * [Instructions](#instructions)
+    * [Defining Column Names Field-Specific Instructions](#defining-column-names-field-specific-instructions)
+    * [Prompting Structure](#prompting-structure)
+    * [Mapping Columns for VoucherVisionEditor](#mapping-columns-for-vouchervisioneditor)
+* [Expense Reporting](#expense-reporting)
+    * [Expense Report Dashboard](#expense-report-dashboard)
+* [User Interface Images](#user-interface-images)
+
+---
+
+# About
+## **VoucherVision** - In Beta Testing Phase 🚀
+
+For inquiries, feedback (or if you want to get involved!) [please complete our form](https://docs.google.com/forms/d/e/1FAIpQLSe2E9zU1bPJ1BW4PMakEQFsRmLbQ0WTBI2UXHIMEFm4WbnAVw/viewform?usp=sf_link).
+
+## **Overview:**  
+Initiated by the **University of Michigan Herbarium**, VoucherVision harnesses the power of large language models (LLMs) to transform the transcription process of natural history specimen labels. Our workflow is as follows:
+- Text extraction from specimen labels with **LeafMachine2**.
+- Text interpretation using **Google Vision OCR**.
+- LLMs, including ***GPT-3.5***, ***GPT-4***, ***PaLM 2***, and Azure instances of OpenAI models, standardize the OCR output into a consistent spreadsheet format. This data can then be integrated into various databases like Specify, Symbiota, and BRAHMS.
+  
+For ensuring accuracy and consistency, the [VoucherVisionEditor](https://github.com/Gene-Weaver/VoucherVisionEditor) serves as a quality control tool.
+
+## Roadmap and New Features List
+
+#### VoucherVision
+- [X] Update to GPT 1106 builds
+- [ ] Option to zip output files for simpler import into VVE
+- [ ] Instead of saving a copy of the original image inplace of the OCR/collage images when they are not selected, just change the path to the original image.
+- [x] Expense tracking
+    - [x] Dashboard
+    - [X] More granular support for different GPT versions
+    - [x] Project-based and cummulative tracking
+- [x] Hugging Face Spaces
+    - [x] Working and refactored
+    - [ ] Visualize locations on a map (verbatim and decimal)
+    - [x] Tested with batch of 300 images
+        - [x] GPT 3.5
+        - [ ] GPT 4
+        - [ ] PaLM 2
+    - [ ] Optimize for +300 images at a time
+- [x] Modular Prompt Builder
+    - [x] Build, save, load, submit to VV library
+    - [ ] Assess whether order of column matters
+    - [ ] Assess shorter prompt effectiveness
+    - [ ] Restrict special columns to conform with VVE requirements (catalog_number, coordinates)
+- [ ] Option to load existing OCR into VoucherVision workflow
+#### Supported LLM APIs
+- [x] OpenAI 
+    - [x] GPT 4
+    - [x] GPT 4 Turbo 1106-preview
+    - [x] GPT 4 32k
+    - [x] GPT 3.5
+    - [x] GPT 3.5 Instruct
+- [x] OpenAI (Microsoft Azure Endpoints)
+    - [x] GPT 4
+    - [x] GPT 4 Turbo 1106-preview
+    - [x] GPT 4 32k
+    - [x] GPT 3.5
+    - [x] GPT 3.5 Instruct
+- [x] MistralAI
+    - [x] Mistral Tiny
+    - [x] Mistral Small
+    - [x] Mistral Medium
+- [x] Google PaLM2
+    - [x] text-bison@001
+    - [x] text-bison@002
+    - [x] text-unicorn@001
+- [x] Google Gemini
+    - [x] Gemini-Pro
+#### Supported Locally Hosted LLMs
+- [x] MistralAI (24GB+ VRAM GPU Required)
+    - [x] Mixtral 8x7B Instruct v0.1
+    - [x] Mixtral 7B Instruct v0.2
+- [x] MistralAI (CPU Inference) ((can run on almost computer!))
+    - [x] Mixtral 7B Instruct v0.2 GGUF via llama.cpp
+- [x] Meta-Llama2 7B
+    - [ ] Llama2 7B chat hf 
+
+#### VoucherVisionEditor
+- [ ] Streamline the startup procedure
+- [ ] Add configurable dropdown menus for certain fields
+- [ ] Make sure that VVE can accomodate arbitrary column names
+- [ ] Remove legacy support (version 1 prompts)
+- [ ] Taxonomy validation helper
+- [x] Visualize locations on a map (verbatim and decimal)
+- [ ] More support for datum and verbatim coordinates
+- [ ] Compare raw OCR to values in form to flag hallucinations/generated content
+- [ ] Accept zipped folders as input
+- [ ] Flag user when multiple people/names/determinations are present
+
+### **Package Information:**  
+The main VoucherVision tool and the VoucherVisionEditor are packaged separately. This separation ensures that lower-performance computers can still install and utilize the editor. While VoucherVision is optimized to function smoothly on virtually any modern system, maximizing its capabilities (like using LeafMachine2 label collages or running Retrieval Augmented Generation (RAG) prompts) mandates a GPU.
+
+> ***NOTE:*** You can absolutely run VoucherVision on non-GPU systems, but RAG will not be possible (luckily the apparent best prompts 'Version2+' does not use RAG). 
+
+---
+
+# Try our public demo!
+Our public demo, while lacking several quality control and reliability features found in the full VoucherVision module, provides an exciting glimpse into its capabilities. Feel free to upload your herbarium specimen and see what happens!
+[VoucherVision Demo](https://huggingface.co/spaces/phyloforfun/VoucherVision)
+
+---
+
+# Installing VoucherVision
+
+## Prerequisites
+- Python 3.10 or later 
+- Optional: an Nvidia GPU + CUDA for running LeafMachine2
+
+## Installation - Cloning the VoucherVision Repository
+1. First, install Python 3.10, or greater, on your machine of choice. We have validated up to Python 3.11.
+    - Make sure that you can use `pip` to install packages on your machine, or at least inside of a virtual environment.
+    - Simply type `pip` into your terminal or PowerShell. If you see a list of options, you are all set. Otherwise, see
+    either this [PIP Documentation](https://pip.pypa.io/en/stable/installation/) or [this help page](https://www.geeksforgeeks.org/how-to-install-pip-on-windows/)
+2. Open a terminal window and `cd` into the directory where you want to install VoucherVision.
+3. In the [Git BASH terminal](https://gitforwindows.org/), clone the VoucherVision repository from GitHub by running the command:
+    <pre><code class="language-python">git clone https://github.com/Gene-Weaver/VoucherVision.git</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+4. Move into the VoucherVision directory by running `cd VoucherVision` in the terminal.
+5. To run VoucherVision we need to install its dependencies inside of a python virtual environmnet. Follow the instructions below for your operating system. 
+
+## About Python Virtual Environments
+A virtual environment is a tool to keep the dependencies required by different projects in separate places, by creating isolated python virtual environments for them. This avoids any conflicts between the packages that you have installed for different projects. It makes it easier to maintain different versions of packages for different projects.
+
+For more information about virtual environments, please see [Creation of virtual environments](https://docs.python.org/3/library/venv.html)
+
+---
+
+## Installation - Windows 10+
+Installation should basically be the same for Linux.
+### Virtual Environment
+
+1. Still inside the VoucherVision directory, show that a venv is currently not active 
+    <pre><code class="language-python">python --version</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+2. Then create the virtual environment (venv_VV is the name of our new virtual environment)  
+    <pre><code class="language-python">python3 -m venv venv_VV</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+    Or depending on your Python version...
+    <pre><code class="language-python">python -m venv venv_VV</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+3. Activate the virtual environment  
+    <pre><code class="language-python">.\venv_VV\Scripts\activate</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+4. Confirm that the venv is active (should be different from step 1)  
+    <pre><code class="language-python">python --version</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+5. If you want to exit the venv later for some reason, deactivate the venv using  
+    <pre><code class="language-python">deactivate</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+
+### Installing Packages
+
+1. Install the required dependencies to use VoucherVision  
+    - Option A - If you are using Windows PowerShell:
+    <pre><code class="language-python">pip install wheel streamlit streamlit-extras plotly pyyaml Pillow pandas matplotlib matplotlib-inline tqdm openai langchain tiktoken openpyxl google-generativeai google-cloud-storage google-cloud-vision opencv-python chromadb chroma-migrate InstructorEmbedding transformers sentence-transformers seaborn dask psutil py-cpuinfo azureml-sdk azure-identity ; if ($?) { pip install numpy -U } ; if ($?) { pip install -U scikit-learn } ; if ($?) { pip install --upgrade numpy scikit-learnstreamlit google-generativeai google-cloud-storage google-cloud-vision azureml-sdk azure-identity openai langchain }</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+
+    - Option B:
+    <pre><code class="language-python">pip install wheel streamlit streamlit-extras plotly pyyaml Pillow pandas matplotlib matplotlib-inline tqdm openai langchain tiktoken openpyxl google-generativeai google-cloud-storage google-cloud-vision opencv-python chromadb chroma-migrate InstructorEmbedding transformers sentence-transformers seaborn dask psutil py-cpuinfo azureml-sdk azure-identity</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+
+2. Upgrade important packages. Run this if there is an update to VoucherVision.
+    <pre><code class="language-python">pip install --upgrade numpy scikit-learn streamlit google-generativeai google-cloud-storage google-cloud-vision azureml-sdk azure-identity openai langchain</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+
+3. Install PyTorch
+    - The LeafMachine2 machine learning algorithm requires PyTorch. If your computer does not have a GPU, then please install a version of PyTorch that is for CPU only. If your computer does have an Nvidia GPU, then please determine which version of PyTorch matches your current CUDA version. Please see [Troubleshooting CUDA](#troubleshooting-cuda) for help. PyTorch is large and will take a bit to install.
+
+    - WITH GPU (or visit [PyTorch.org](https://pytorch.org/get-started/locally/) to find the appropriate version of PyTorch for your CUDA version)
+    <pre><code class="language-python">pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+    - WITHOUT GPU, CPU ONLY
+    <pre><code class="language-python">pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cpu</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+    
+
+> If you need help, please submit an inquiry in the form at [LeafMachine.org](https://LeafMachine.org/)
+
+---
+
+## Troubleshooting CUDA
+
+- If your system already has another version of CUDA (e.g., CUDA 11.7) then it can be complicated to switch to CUDA 11.3. 
+- The simplest solution is to install pytorch with CPU only, avoiding the CUDA problem entirely.
+- Alternatively, you can install the [latest pytorch release](https://pytorch.org/get-started/locally/) for your specific system, either using the cpu only version `pip3 install torch`, `pip3 install torchvision`, `pip3 install torchaudio` or by matching the pythorch version to your CUDA version.
+- We have not validated CUDA 11.6 or CUDA 11.7, but our code is likely to work with them too. If you have success with other versions of CUDA/pytorch, let us know and we will update our instructions. 
+
+---
+
+# Create a Desktop Shortcut to Launch VoucherVision GUI
+We can create a desktop shortcut to launch VoucherVision. In the `../VoucherVision/` directory is a file called `create_desktop_shortcut.py`. In the terminal, move into the `../VoucherVision/` directory and type:
+<pre><code class="language-python">python create_desktop_shortcut.py</code></pre>
+<button class="btn" data-clipboard-target="#code-snippet"></button>
+Or...
+<pre><code class="language-python">python3 create_desktop_shortcut.py</code></pre>
+<button class="btn" data-clipboard-target="#code-snippet"></button>
+Follow the instructions, select where you want the shortcut to be created, then where the virtual environment is located. 
+
 ---
-title: VoucherVision
-emoji: 📈
-colorFrom: blue
-colorTo: green
-sdk: streamlit
-sdk_version: 1.28.1
-app_file: app.py
-pinned: false
-license: cc-by-nc-4.0
+
+# Run VoucherVision
+1. In the terminal, make sure that you `cd` into the `VoucherVision` directory and that your virtual environment is active (you should see venv_VV on the command line). 
+2. Type:
+    <pre><code class="language-python">python run_VoucherVision.py</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+    or depending on your Python installation:
+    <pre><code class="language-python">python3 run_VoucherVision.py</code></pre>
+    <button class="btn" data-clipboard-target="#code-snippet"></button>
+3. If you ever see an error that says that a "port is not available", open `run_VoucherVision.py` in a plain text editor and change the `--port` value to something different but close, like 8502.
+
+## Setting up API key
+VoucherVision requires access to Google Vision OCR and at least one of the following LLMs: OpenAI API, Google PaLM 2, a private instance of OpenAI through Microsoft Azure. On first startup, you will see a page with instructions on how to get these API keys. ***Nothing will work until*** you get at least the Google Vision OCR API key and one LLM API key. 
+
+## Check GPU
+Press the "Check GPU" button to see if you have a GPU available. If you know that your computer has an Nvidia GPU, but the check fails, then you need to install an different version of PyTorch in the virtual environment. 
+
+## Run Tests
+Once you have provided API keys, you can test all available prompts and LLMs by pressing the test buttons. Every combination of LLM, prompt, and LeafMachine2 collage will run on the image in the `../VoucherVision/demo/demo_images` folder. A grid will appear letting you know which combinations are working on your system. 
+
+## Starting VoucherVision
+1. "Run name" - Set a run name for your project. This will be the name of the new folder that contains the output files.
+2. "Output directory" - Paste the full file path of where you would like to save the folder that will be created in step 1.
+3. "Input images directory" - Paste the full file path of where the input images are located. This folder can only have JPG or JPEG images inside of it.
+4. "Select an LLM" - Pick the LLM you want to use to parse the unstructured OCR text. 
+    - As of Nov. 1, 2023 PaLM 2 is free to use. 
+5. "Prompt Version" - Pick your prompt version. We recommend "Version 2" for production use, but you can experiment with our other prompts. 
+6. "Cropped Components" - Check the box to use LeafMachine2 collage images as the input file. LeafMachine2 can often find small handwritten text that may be missed by Google Vision OCR's text detection algorithm. But, the difference in performance is not that big. You will still get good performance without using the LeafMachine2 collage images.  
+7. "Domain Knowledge" is only used for "Version 1" prompts.
+8. "Component Detector" sets basic LeafMachine2 parameters, but the default is likely good enough.
+9. "Processing Options"
+    - The image file name defines the row name in the final output spreadsheet.
+    - We provide some basic options to clean/parse the image file name to produce the desired output. 
+    - For example, if the input image name is `MICH-V-3819482.jpg` but the desired name is just `3819482` you can add `MICH-V-` to the "Remove prefix from catalog number" input box. Alternatively, you can check the "Require Catalog..." box and achieve the same result. 
+
+10. ***Finally*** you can press the start processing button.
+
+## Azure Instances of OpenAI
+If your institution has an enterprise instance of OpenAI's services, [like at the University of Michigan](https://its.umich.edu/computing/ai), you can use Azure instead of the OpenAI servers. Your institution should be able to provide you with the required keys (there are 5 required keys for this service). 
+
+# Custom Prompt Builder
+VoucherVision empowers individual institutions to customize the format of the LLM output. Using our pre-defined prompts you can transcribe the label text into 20 columns, but using our Prompt Builder you can load one of our default prompts and adjust the output to meet your needs. More instructions will come soon, but for now here are a few more details. 
+
+### Load, Build, Edit
+
+The Prompt Builder creates a prompt in the structure that VoucherVision expects. This information is stored as a configuration yaml file in `../VoucherVision/custom_prompts/`. We provide a few versions to get started. You can load one of our examples and then use the Prompt Builder to edit or add new columns. 
+
+![prompt_1](https://LeafMachine.org/img/prompt_1.PNG)
+
+### Instructions
+
+Right now, the prompting instructions are not configurable, but that may change in the future. 
+
+![prompt_2](https://LeafMachine.org/img/prompt_1.PNG)
+
+### Defining Column Names Field-Specific Instructions
+
+The central JSON object shows the structure of the columns that you are requesting the LLM to create and populate with information from the specimen's labels. These will become the rows in the final xlsx file the VoucherVision generates. You can pick formatting instructions, set default values, and give detailed instructions.
+
+> Note: formatting instructions are not always followed precisely by the LLM. For example, GPT-4 is capable of granular instructions like converting ALL CAPS TEXT to sentence-case, but GPT-3.5 and PaLM 2 might not be capable of following that instruction every time (which is why we have the VoucherVisionEditor and are working to link these instructions so that humans editing the output can quickly/easily fix these errors).
+
+![prompt_3](https://LeafMachine.org/img/prompt_3.PNG)
+
+### Prompting Structure
+
+The rightmost JSON object is the entire prompt structure. If you load the `required_structure.yaml` prompt, you will wee the bare-bones version of what VoucherVision expects to see. All of the parts are there for a reason. The Prompt Builder UI may be a little unruly right now thanks to quirks with Streamlit, but we still recommend using the UI to build your own prompts to make sure that all of the required components are present. 
+
+![prompt_4](https://LeafMachine.org/img/prompt_4.PNG)
+
+### Mapping Columns for VoucherVisionEditor
+
+Finally, we need to map columns to a VoucherVisionEditor category.
+
+![prompt_5](https://LeafMachine.org/img/prompt_5.PNG)
+
+# Expense Reporting
+VoucherVision logs the number of input and output tokens (using [tiktoken](https://github.com/openai/tiktoken)) from every call. We store the publicly listed prices of the LLM APIs in `../VoucherVision/api_cost/api_cost.yaml`. Then we do some simple math to estimage the cost of run, which is stored inside of your project's output directory `../run_name/Cost/run_name.csv` and all runs are accumulated in a csv file stored in `../VoucherVision/expense_report/expense_report.csv`. VoucherVision only manages `expense_report.csv`, so if you want to split costs by month/quarter then copy and rename `expense_report.csv`. Deleting `expense_report.csv` will let you accumulate more stats.  
+
+> This should be treated as an estimate. The true cost may be slightly different.  
+
+This is an example of the stats that we track:
+| run                        | date                     | api_version | total_cost | n_images | tokens_in | tokens_out | rate_in | rate_out | cost_in   | cost_out |
+|----------------------------|--------------------------|-------------|------------|----------|-----------|------------|---------|----------|-----------|----------|
+| GPT4_test_run1  | 2023_11_05__17-44-31     | GPT_4       | 0.23931    | 2        | 6749      | 614        | 0.03    | 0.06     | 0.20247   | 0.03684  |
+| GPT_3_5_test_run  | 2023_11_05__17-48-48     | GPT_3_5     | 0.0189755  | 4        | 12033     | 463        | 0.0015  | 0.002    | 0.0180495 | 0.000926 |
+| PALM2_test_run  | 2023_11_05__17-50-35     | PALM2       | 0          | 4        | 13514     | 771        | 0       | 0        | 0         | 0        |
+| GPT4_test_run2  | 2023_11_05__18-49-24     | GPT_4       | 0.40962    | 4        | 12032     | 811        | 0.03    | 0.06     | 0.36096   | 0.04866  |
+
+## Expense Report Dashboard
+The sidebar in VoucherVision displays summary stats taken from `expense_report.csv`.
+![Expense Report Dashboard](https://LeafMachine.org/img/expense_report.PNG)
+
+# User Interface Images
+Validation test when the OpenAI key is not provided, but keys for PaLM 2 and Azure OpenAI are present:
+![Validation 1](https://LeafMachine.org/img/validation_1.PNG)
+
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+Validation test when all versions of the OpenAI keys are provided:
+![Validation GPT](https://LeafMachine.org/img/validation_gpt.PNG)
+
+---
+
+A successful GPU test:
+![Validation GPU](https://LeafMachine.org/img/validation_gpu.PNG)
+
+---
+
+Successful PaLM 2 test:
+![Validation PaLM](https://LeafMachine.org/img/validation_palm.PNG)
+
+
+
+

api_cost/api_cost.yaml

@@ -1,9 +1,82 @@
+# OpenAI
+# https://openai.com/pricing
+
+GPT_4_32K:
+  in: 0.06
+  out: 0.12
+GPT_4:
+  in: 0.03
+  out: 0.06
+GPT_4_TURBO:
+  in: 0.01
+  out: 0.03
+GPT_3_5_INSTRUCT:
+  in: 0.0015
+  out: 0.0020
 GPT_3_5:
   in: 0.0010
   out: 0.0020
-GPT_4:
+
+# Azure
+AZURE_GPT_4_32K:
+  in: 0.06
+  out: 0.12
+AZURE_GPT_4:
+  in: 0.03
+  out: 0.06
+AZURE_GPT_4_TURBO:
   in: 0.01
   out: 0.03
-PALM2:
+AZURE_GPT_3_5_INSTRUCT:
+  in: 0.0015
+  out: 0.0020
+AZURE_GPT_3_5:
+  in: 0.0010
+  out: 0.0020
+
+
+# Google Gemini
+# https://ai.google.dev/pricing
+GEMINI_PRO:
+  in: 0.00025
+  out: 0.0005
+
+
+# Google PaLM 2 (text-bison, text-unicorn)
+# https://cloud.google.com/vertex-ai/docs/generative-ai/pricing
+PALM2_TU_1:
+  in: 0.00025
+  out: 0.00020
+PALM2_TB_1:
+  in: 0.00025
+  out: 0.0005
+PALM2_TB_2:
+  in: 0.00025
+  out: 0.0005
+
+
+# Mistral AI 
+# https://docs.mistral.ai/platform/pricing/
+MISTRAL_MEDIUM:
+  in: 0.00250
+  out: 0.00750
+MISTRAL_SMALL:
+  in: 0.00060
+  out: 0.00180
+MISTRAL_TINY:
+  in: 0.00014
+  out: 0.00042
+
+
+################
+# Local Models
+################
+LOCAL_MIXTRAL_8X7B_INSTRUCT_V01:
   in: 0.0
   out: 0.0
+LOCAL_MISTRAL_7B_INSTRUCT_V02: 
+  in: 0.0
+  out: 0.0
+LOCAL_CPU_MISTRAL_7B_INSTRUCT_V02_GGUF:
+  in: 0.0
+  out: 0.0

custom_prompts/required_structure.yaml

@@ -1,65 +0,0 @@
-prompt_author: unknown          
-prompt_author_institution: unknown          
-prompt_description: unknown          
-LLM: gpt
-instructions: '1. Refactor the unstructured OCR text into a dictionary based on the
-  JSON structure outlined below.
-
-  2. You should map the unstructured OCR text to the appropriate JSON key and then
-  populate the field based on its rules.
-
-  3. Some JSON key fields are permitted to remain empty if the corresponding information
-  is not found in the unstructured OCR text.
-
-  4. Ignore any information in the OCR text that doesn''t fit into the defined JSON
-  structure.
-
-  5. Duplicate dictionary fields are not allowed.
-
-  6. Ensure that all JSON keys are in lowercase.
-
-  7. Ensure that new JSON field values follow sentence case capitalization.
-
-  8. Ensure all key-value pairs in the JSON dictionary strictly adhere to the format
-  and data types specified in the template.
-
-  9. Ensure the output JSON string is valid JSON format. It should not have trailing
-  commas or unquoted keys.
-
-  10. Only return a JSON dictionary represented as a string. You should not explain
-  your answer.'
-json_formatting_instructions: "The next section of instructions outlines how to format\
-  \ the JSON dictionary. The keys are the same as those of the final formatted JSON\
-  \ object.\nFor each key there is a format requirement that specifies how to transcribe\
-  \ the information for that key. \nThe possible formatting options are:\n1. \"verbatim\
-  \ transcription\" - field is populated with verbatim text from the unformatted OCR.\n\
-  2. \"spell check transcription\" - field is populated with spelling corrected text\
-  \ from the unformatted OCR.\n3. \"boolean yes no\" - field is populated with only\
-  \ yes or no.\n4. \"boolean 1 0\" - field is populated with only 1 or 0.\n5. \"integer\"\
-  \ - field is populated with only an integer.\n6. \"[list]\" - field is populated\
-  \ from one of the values in the list.\n7. \"yyyy-mm-dd\" - field is populated with\
-  \ a date in the format year-month-day.\nThe desired null value is also given. Populate\
-  \ the field with the null value of the information for that key is not present in\
-  \ the unformatted OCR text."
-mapping:
-  # Add column names to the desired category. This is used to map the VV Editor.
-  COLLECTING: []
-  GEOGRAPHY: []
-  LOCALITY: []
-  MISCELLANEOUS: []
-  TAXONOMY:
-  - catalog_number
-rules:
-  Dictionary:
-    # Manually add rows here. You MUST keep 'catalog_number' unchanged. Use 'catalog_number' as a guide for adding more columns.
-    # The only values allowed in the 'format' key are those outlines above in the 'json_formatting_instructions' section.
-    # If you want an empty cell by default, use '' for the 'null_value'.
-    catalog_number:
-      description: The barcode identifier, typically a number with at least 6 digits,
-        but fewer than 30 digits.
-      format: verbatim transcription
-      null_value: ''
-  # Do not change or remove below. This is required for some LLMs
-  SpeciesName:
-    taxonomy:
-    - Genus_species

custom_prompts/version_2.yaml

@@ -1,232 +0,0 @@
-prompt_author: Will Weaver          
-prompt_author_institution: UM          
-prompt_description: Basic prompt used by the University of Michigan. Designed to be a starting point for more complex prompts.   
-LLM: gpt
-instructions: '1. Refactor the unstructured OCR text into a dictionary based on the
-  JSON structure outlined below.
-
-  2. You should map the unstructured OCR text to the appropriate JSON key and then
-  populate the field based on its rules.
-
-  3. Some JSON key fields are permitted to remain empty if the corresponding information
-  is not found in the unstructured OCR text.
-
-  4. Ignore any information in the OCR text that doesn''t fit into the defined JSON
-  structure.
-
-  5. Duplicate dictionary fields are not allowed.
-
-  6. Ensure that all JSON keys are in lowercase.
-
-  7. Ensure that new JSON field values follow sentence case capitalization.
-
-  8. Ensure all key-value pairs in the JSON dictionary strictly adhere to the format
-  and data types specified in the template.
-
-  9. Ensure the output JSON string is valid JSON format. It should not have trailing
-  commas or unquoted keys.
-
-  10. Only return a JSON dictionary represented as a string. You should not explain
-  your answer.'
-json_formatting_instructions: "The next section of instructions outlines how to format\
-  \ the JSON dictionary. The keys are the same as those of the final formatted JSON\
-  \ object.\nFor each key there is a format requirement that specifies how to transcribe\
-  \ the information for that key. \nThe possible formatting options are:\n1. \"verbatim\
-  \ transcription\" - field is populated with verbatim text from the unformatted OCR.\n\
-  2. \"spell check transcription\" - field is populated with spelling corrected text\
-  \ from the unformatted OCR.\n3. \"boolean yes no\" - field is populated with only\
-  \ yes or no.\n4. \"boolean 1 0\" - field is populated with only 1 or 0.\n5. \"integer\"\
-  \ - field is populated with only an integer.\n6. \"[list]\" - field is populated\
-  \ from one of the values in the list.\n7. \"yyyy-mm-dd\" - field is populated with\
-  \ a date in the format year-month-day.\nThe desired null value is also given. Populate\
-  \ the field with the null value of the information for that key is not present in\
-  \ the unformatted OCR text."
-mapping:
-  COLLECTING:
-  - collectors
-  - collector_number
-  - determined_by
-  - multiple_names
-  - verbatim_date
-  - date
-  - end_date
-  GEOGRAPHY:
-  - country
-  - state
-  - county
-  - min_elevation
-  - max_elevation
-  - elevation_units
-  LOCALITY:
-  - locality_name
-  - verbatim_coordinates
-  - decimal_coordinates
-  - datum
-  - plant_description
-  - cultivated
-  - habitat
-  MISCELLANEOUS: []
-  TAXONOMY:
-  - catalog_number
-  - genus
-  - species
-  - subspecies
-  - variety
-  - forma
-rules:
-  Dictionary:
-    catalog_number:
-      description: The barcode identifier, typically a number with at least 6 digits,
-        but fewer than 30 digits.
-      format: verbatim transcription
-      null_value: ''
-    collector_number:
-      description: Unique identifier or number that denotes the specific collecting
-        event and associated with the collector.
-      format: verbatim transcription
-      null_value: s.n.
-    collectors:
-      description: Full name(s) of the individual(s) responsible for collecting the
-        specimen. When multiple collectors are involved, their names should be separated
-        by commas.
-      format: verbatim transcription
-      null_value: not present
-    country:
-      description: Country that corresponds to the current geographic location of
-        collection. Capitalize first letter of each word. If abbreviation is given
-        populate field with the full spelling of the country's name.
-      format: spell check transcription
-      null_value: ''
-    county:
-      description: Administrative division 2 that corresponds to the current geographic
-        location of collection; capitalize first letter of each word. Administrative
-        division 2 is equivalent to a U.S. county, parish, borough.
-      format: spell check transcription
-      null_value: ''
-    cultivated:
-      description: Cultivated plants are intentionally grown by humans. In text descriptions,
-        look for planting dates, garden locations, ornamental, cultivar names, garden,
-        or farm to indicate cultivated plant.
-      format: boolean yes no
-      null_value: ''
-    date:
-      description: 'Date the specimen was collected formatted as year-month-day. If
-        specific components of the date are unknown, they should be replaced with
-        zeros. Examples: ''0000-00-00'' if the entire date is unknown, ''YYYY-00-00''
-        if only the year is known, and ''YYYY-MM-00'' if year and month are known
-        but day is not.'
-      format: yyyy-mm-dd
-      null_value: ''
-    datum:
-      description: Datum of location coordinates. Possible values are include in the
-        format list. Leave field blank if unclear. [WGS84, WGS72, WGS66, WGS60, NAD83,
-        NAD27, OSGB36, ETRS89, ED50, GDA94, JGD2011, Tokyo97, KGD2002, TWD67, TWD97,
-        BJS54, XAS80, GCJ-02, BD-09, PZ-90.11, GTRF, CGCS2000, ITRF88, ITRF89, ITRF90,
-        ITRF91, ITRF92, ITRF93, ITRF94, ITRF96, ITRF97, ITRF2000, ITRF2005, ITRF2008,
-        ITRF2014, Hong Kong Principal Datum, SAD69]
-      format: '[list]'
-      null_value: ''
-    decimal_coordinates:
-      description: Correct and convert the verbatim location coordinates to conform
-        with the decimal degrees GPS coordinate format.
-      format: spell check transcription
-      null_value: ''
-    determined_by:
-      description: Full name of the individual responsible for determining the taxanomic
-        name of the specimen. Sometimes the name will be near to the characters 'det'
-        to denote determination. This name may be isolated from other names in the
-        unformatted OCR text.
-      format: verbatim transcription
-      null_value: ''
-    elevation_units:
-      description: 'Elevation units must be meters. If min_elevation field is populated,
-        then elevation_units: ''m''. Otherwise elevation_units: ''''.'
-      format: spell check transcription
-      null_value: ''
-    end_date:
-      description: 'If a date range is provided, this represents the later or ending
-        date of the collection period, formatted as year-month-day. If specific components
-        of the date are unknown, they should be replaced with zeros. Examples: ''0000-00-00''
-        if the entire end date is unknown, ''YYYY-00-00'' if only the year of the
-        end date is known, and ''YYYY-MM-00'' if year and month of the end date are
-        known but the day is not.'
-      format: yyyy-mm-dd
-      null_value: ''
-    forma:
-      description: Taxonomic determination to form (f.).
-      format: verbatim transcription
-      null_value: ''
-    genus:
-      description: Taxonomic determination to genus. Genus must be capitalized. If
-        genus is not present use the taxonomic family name followed by the word 'indet'.
-      format: verbatim transcription
-      null_value: ''
-    habitat:
-      description: Description of a plant's habitat or the location where the specimen
-        was collected. Ignore descriptions of the plant itself.
-      format: verbatim transcription
-      null_value: ''
-    locality_name:
-      description: Description of geographic location, landscape, landmarks, regional
-        features, nearby places, or any contextual information aiding in pinpointing
-        the exact origin or site of the specimen.
-      format: verbatim transcription
-      null_value: ''
-    max_elevation:
-      description: Maximum elevation or altitude in meters. If only one elevation
-        is present, then max_elevation should be set to the null_value. Only if units
-        are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m'
-        or 'm.' or 'meters'). Round to integer.
-      format: integer
-      null_value: ''
-    min_elevation:
-      description: Minimum elevation or altitude in meters. Only if units are explicit
-        then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or
-        'meters'). Round to integer.
-      format: integer
-      null_value: ''
-    multiple_names:
-      description: Indicate whether multiple people or collector names are present
-        in the unformatted OCR text. If you see more than one person's name the value
-        is 'yes'; otherwise the value is 'no'.
-      format: boolean yes no
-      null_value: ''
-    plant_description:
-      description: Description of plant features such as leaf shape, size, color,
-        stem texture, height, flower structure, scent, fruit or seed characteristics,
-        root system type, overall growth habit and form, any notable aroma or secretions,
-        presence of hairs or bristles, and any other distinguishing morphological
-        or physiological characteristics.
-      format: verbatim transcription
-      null_value: ''
-    species:
-      description: Taxonomic determination to species, do not capitalize species.
-      format: verbatim transcription
-      null_value: ''
-    state:
-      description: Administrative division 1 that corresponds to the current geographic
-        location of collection. Capitalize first letter of each word. Administrative
-        division 1 is equivalent to a U.S. State.
-      format: spell check transcription
-      null_value: ''
-    subspecies:
-      description: Taxonomic determination to subspecies (subsp.).
-      format: verbatim transcription
-      null_value: ''
-    variety:
-      description: Taxonomic determination to variety (var).
-      format: verbatim transcription
-      null_value: ''
-    verbatim_coordinates:
-      description: Verbatim location coordinates as they appear on the label. Do not
-        convert formats. Possible coordinate types are one of [Lat, Long, UTM, TRS].
-      format: verbatim transcription
-      null_value: ''
-    verbatim_date:
-      description: Date of collection exactly as it appears on the label. Do not change
-        the format or correct typos.
-      format: verbatim transcription
-      null_value: s.d.
-  SpeciesName:
-    taxonomy:
-    - Genus_species

custom_prompts/version_2_OSU.yaml

@@ -1,233 +0,0 @@
-prompt_author: Will Weaver          
-prompt_author_institution: UM          
-prompt_description: Version 2 slightly modified for OSU, but still unfinished   
-LLM: gpt
-instructions: '1. Refactor the unstructured OCR text into a dictionary based on the
-  JSON structure outlined below.
-
-  2. You should map the unstructured OCR text to the appropriate JSON key and then
-  populate the field based on its rules.
-
-  3. Some JSON key fields are permitted to remain empty if the corresponding information
-  is not found in the unstructured OCR text.
-
-  4. Ignore any information in the OCR text that doesn''t fit into the defined JSON
-  structure.
-
-  5. Duplicate dictionary fields are not allowed.
-
-  6. Ensure that all JSON keys are in lowercase.
-
-  7. Ensure that new JSON field values follow sentence case capitalization.
-
-  8. Ensure all key-value pairs in the JSON dictionary strictly adhere to the format
-  and data types specified in the template.
-
-  9. Ensure the output JSON string is valid JSON format. It should not have trailing
-  commas or unquoted keys.
-
-  10. Only return a JSON dictionary represented as a string. You should not explain
-  your answer.'
-json_formatting_instructions: "The next section of instructions outlines how to format\
-  \ the JSON dictionary. The keys are the same as those of the final formatted JSON\
-  \ object.\nFor each key there is a format requirement that specifies how to transcribe\
-  \ the information for that key. \nThe possible formatting options are:\n1. \"verbatim\
-  \ transcription\" - field is populated with verbatim text from the unformatted OCR.\n\
-  2. \"spell check transcription\" - field is populated with spelling corrected text\
-  \ from the unformatted OCR.\n3. \"boolean yes no\" - field is populated with only\
-  \ yes or no.\n4. \"boolean 1 0\" - field is populated with only 1 or 0.\n5. \"integer\"\
-  \ - field is populated with only an integer.\n6. \"[list]\" - field is populated\
-  \ from one of the values in the list.\n7. \"yyyy-mm-dd\" - field is populated with\
-  \ a date in the format year-month-day.\nThe desired null value is also given. Populate\
-  \ the field with the null value of the information for that key is not present in\
-  \ the unformatted OCR text."
-mapping:
-  COLLECTING:
-  - collectors
-  - collector_number
-  - determined_by
-  - multiple_names
-  - verbatim_date
-  - date
-  - end_date
-  GEOGRAPHY:
-  - country
-  - state
-  - county
-  - min_elevation
-  - max_elevation
-  - elevation_units
-  LOCALITY:
-  - locality_name
-  - verbatim_coordinates
-  - decimal_coordinates
-  - datum
-  - plant_description
-  - cultivated
-  - habitat
-  MISCELLANEOUS: []
-  TAXONOMY:
-  - catalog_number
-  - genus
-  - species
-  - subspecies
-  - variety
-  - forma
-rules:
-  Dictionary:
-    catalog_number:
-      description: The barcode identifier, typically a number with at least 6 digits,
-        but fewer than 30 digits.
-      format: verbatim transcription
-      null_value: ''
-    collector_number:
-      description: Unique identifier or number that denotes the specific collecting
-        event and associated with the collector.
-      format: verbatim transcription
-      null_value: s.n.
-    collectors:
-      description: Full name(s) of the individual(s) responsible for collecting the
-        specimen. When multiple collectors are involved, their names should be separated
-        by commas.
-      format: verbatim transcription
-      null_value: not present
-    country:
-      description: Country that corresponds to the current geographic location of
-        collection. Capitalize first letter of each word. If abbreviation is given
-        populate field with the full spelling of the country's name.
-      format: spell check transcription
-      null_value: ''
-    county:
-      description: Administrative division 2 that corresponds to the current geographic
-        location of collection; capitalize first letter of each word. Administrative
-        division 2 is equivalent to a U.S. county, parish, borough.
-      format: spell check transcription
-      null_value: ''
-    cultivated:
-      description: Cultivated plants are intentionally grown by humans. In text descriptions,
-        look for planting dates, garden locations, ornamental, cultivar names, garden,
-        or farm to indicate cultivated plant. The value 1 indicates that the specimen
-        was cultivated, the value zero otherwise.
-      format: boolean 1 0
-      null_value: '0'
-    date:
-      description: 'Date the specimen was collected formatted as year-month-day. If
-        specific components of the date are unknown, they should be replaced with
-        zeros. Examples: ''0000-00-00'' if the entire date is unknown, ''YYYY-00-00''
-        if only the year is known, and ''YYYY-MM-00'' if year and month are known
-        but day is not.'
-      format: yyyy-mm-dd
-      null_value: ''
-    datum:
-      description: Datum of location coordinates. Possible values are include in the
-        format list. Leave field blank if unclear. [WGS84, WGS72, WGS66, WGS60, NAD83,
-        NAD27, OSGB36, ETRS89, ED50, GDA94, JGD2011, Tokyo97, KGD2002, TWD67, TWD97,
-        BJS54, XAS80, GCJ-02, BD-09, PZ-90.11, GTRF, CGCS2000, ITRF88, ITRF89, ITRF90,
-        ITRF91, ITRF92, ITRF93, ITRF94, ITRF96, ITRF97, ITRF2000, ITRF2005, ITRF2008,
-        ITRF2014, Hong Kong Principal Datum, SAD69]
-      format: '[list]'
-      null_value: ''
-    decimal_coordinates:
-      description: Correct and convert the verbatim location coordinates to conform
-        with the decimal degrees GPS coordinate format.
-      format: spell check transcription
-      null_value: ''
-    determined_by:
-      description: Full name of the individual responsible for determining the taxanomic
-        name of the specimen. Sometimes the name will be near to the characters 'det'
-        to denote determination. This name may be isolated from other names in the
-        unformatted OCR text.
-      format: verbatim transcription
-      null_value: ''
-    elevation_units:
-      description: 'Elevation units must be meters. If min_elevation field is populated,
-        then elevation_units: ''m''. Otherwise elevation_units: ''''.'
-      format: spell check transcription
-      null_value: ''
-    end_date:
-      description: 'If a date range is provided, this represents the later or ending
-        date of the collection period, formatted as year-month-day. If specific components
-        of the date are unknown, they should be replaced with zeros. Examples: ''0000-00-00''
-        if the entire end date is unknown, ''YYYY-00-00'' if only the year of the
-        end date is known, and ''YYYY-MM-00'' if year and month of the end date are
-        known but the day is not.'
-      format: yyyy-mm-dd
-      null_value: ''
-    forma:
-      description: Taxonomic determination to form (f.).
-      format: verbatim transcription
-      null_value: ''
-    genus:
-      description: Taxonomic determination to genus. Genus must be capitalized. If
-        genus is not present use the taxonomic family name followed by the word 'indet'.
-      format: verbatim transcription
-      null_value: ''
-    habitat:
-      description: Description of a plant's habitat or the location where the specimen
-        was collected. Ignore descriptions of the plant itself.
-      format: verbatim transcription
-      null_value: ''
-    locality_name:
-      description: Description of geographic location, landscape, landmarks, regional
-        features, nearby places, or any contextual information aiding in pinpointing
-        the exact origin or site of the specimen.
-      format: verbatim transcription
-      null_value: ''
-    max_elevation:
-      description: Maximum elevation or altitude in meters. If only one elevation
-        is present, then max_elevation should be set to the null_value. Only if units
-        are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m'
-        or 'm.' or 'meters'). Round to integer.
-      format: integer
-      null_value: ''
-    min_elevation:
-      description: Minimum elevation or altitude in meters. Only if units are explicit
-        then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or
-        'meters'). Round to integer.
-      format: integer
-      null_value: ''
-    multiple_names:
-      description: Indicate whether multiple people or collector names are present
-        in the unformatted OCR text. If you see more than one person's name the value
-        is 'yes'; otherwise the value is 'no'.
-      format: boolean yes no
-      null_value: ''
-    plant_description:
-      description: Description of plant features such as leaf shape, size, color,
-        stem texture, height, flower structure, scent, fruit or seed characteristics,
-        root system type, overall growth habit and form, any notable aroma or secretions,
-        presence of hairs or bristles, and any other distinguishing morphological
-        or physiological characteristics.
-      format: verbatim transcription
-      null_value: ''
-    species:
-      description: Taxonomic determination to species, do not capitalize species.
-      format: verbatim transcription
-      null_value: ''
-    state:
-      description: Administrative division 1 that corresponds to the current geographic
-        location of collection. Capitalize first letter of each word. Administrative
-        division 1 is equivalent to a U.S. State.
-      format: spell check transcription
-      null_value: ''
-    subspecies:
-      description: Taxonomic determination to subspecies (subsp.).
-      format: verbatim transcription
-      null_value: ''
-    variety:
-      description: Taxonomic determination to variety (var).
-      format: verbatim transcription
-      null_value: ''
-    verbatim_coordinates:
-      description: Verbatim location coordinates as they appear on the label. Do not
-        convert formats. Possible coordinate types are one of [Lat, Long, UTM, TRS].
-      format: verbatim transcription
-      null_value: ''
-    verbatim_date:
-      description: Date of collection exactly as it appears on the label. Do not change
-        the format or correct typos.
-      format: verbatim transcription
-      null_value: s.d.
-  SpeciesName:
-    taxonomy:
-    - Genus_species

run_VoucherVision.py

@@ -23,9 +23,13 @@ if __name__ == "__main__":
     sys.argv = [
         "streamlit",
         "run",
-        resolve_path(os.path.join(dir_home,"vouchervision", "VoucherVision_GUI.py")),
+        resolve_path(os.path.join(dir_home,"app.py")),
+        # resolve_path(os.path.join(dir_home,"vouchervision", "VoucherVision_GUI.py")),
         "--global.developmentMode=false",
-        "--server.port=8525",
-
+        # "--server.port=8545",
+        "--server.port=8546",
+        # Toggle below for HF vs Local
+        "--is_hf=1",
+        # "--is_hf=0",
     ]
     sys.exit(stcli.main())

settings/bottom.yaml

@@ -0,0 +1,82 @@
+leafmachine:
+  LLM_version: Azure GPT 4 Turbo 1106-preview
+  archival_component_detector:
+    detector_iteration: PREP_final
+    detector_type: Archival_Detector
+    detector_version: PREP_final
+    detector_weights: best.pt
+    do_save_prediction_overlay_images: true
+    ignore_objects_for_overlay: []
+    minimum_confidence_threshold: 0.5
+  cropped_components:
+    binarize_labels: false
+    binarize_labels_skeletonize: false
+    do_save_cropped_annotations: true
+    save_cropped_annotations:
+    - label
+    - barcode
+    save_per_annotation_class: true
+    save_per_image: false
+  data:
+    do_apply_conversion_factor: false
+    include_darwin_core_data_from_combined_file: false
+    save_individual_csv_files_landmarks: false
+    save_individual_csv_files_measurements: false
+    save_individual_csv_files_rulers: false
+    save_individual_efd_files: false
+    save_json_measurements: false
+    save_json_rulers: false
+  do:
+    check_for_corrupt_images_make_vertical: true
+    check_for_illegal_filenames: false
+  do_create_OCR_helper_image: false
+  logging:
+    log_level: null
+  modules:
+    specimen_crop: true
+  overlay:
+    alpha_transparency_archival: 0.3
+    alpha_transparency_plant: 0
+    alpha_transparency_seg_partial_leaf: 0.3
+    alpha_transparency_seg_whole_leaf: 0.4
+    ignore_archival_detections_classes: []
+    ignore_landmark_classes: []
+    ignore_plant_detections_classes:
+    - leaf_whole
+    - specimen
+    line_width_archival: 12
+    line_width_efd: 12
+    line_width_plant: 12
+    line_width_seg: 12
+    overlay_background_color: black
+    overlay_dpi: 300
+    save_overlay_to_jpgs: true
+    save_overlay_to_pdf: false
+    show_archival_detections: true
+    show_landmarks: true
+    show_plant_detections: true
+    show_segmentations: true
+  print:
+    optional_warnings: true
+    verbose: true
+  project:
+    OCR_option: both
+    batch_size: 500
+    build_new_embeddings_database: false
+    catalog_numerical_only: false
+    continue_run_from_partial_xlsx: ''
+    delete_all_temps: false
+    delete_temps_keep_VVE: false
+    dir_images_local: d:\Dropbox\VoucherVision\demo\demo_images
+    dir_output: C:\Users\Will\Downloads
+    do_use_trOCR: true
+    embeddings_database_name: SLTP_UM_AllAsiaMinimalInRegion
+    image_location: local
+    num_workers: 8
+    path_to_domain_knowledge_xlsx: d:\Dropbox\VoucherVision\domain_knowledge\SLTP_UM_AllAsiaMinimalInRegion.xlsx
+    prefix_removal: ''
+    prompt_version: SLTPvA_long.yaml
+    run_name: test
+    suffix_removal: ''
+    use_domain_knowledge: false
+  use_RGB_label_images: true

vouchervision/API_validation.py

@@ -0,0 +1,224 @@
+import os, io, openai, vertexai
+from mistralai.client import MistralClient
+from mistralai.models.chat_completion import ChatMessage
+from langchain.schema import HumanMessage
+from langchain_openai import AzureChatOpenAI
+from vertexai.language_models import TextGenerationModel
+from vertexai.preview.generative_models import GenerativeModel
+from google.cloud import vision
+from datetime import datetime
+
+
+
+class APIvalidation:
+
+    def __init__(self, cfg_private, dir_home) -> None:
+        self.cfg_private = cfg_private
+        self.dir_home = dir_home
+        self.formatted_date = self.get_formatted_date()
+
+    def get_formatted_date(self):
+        # Get the current date
+        current_date = datetime.now()
+
+        # Format the date as "Month day, year" (e.g., "January 23, 2024")
+        formatted_date = current_date.strftime("%B %d, %Y")
+
+        return formatted_date
+
+
+    def has_API_key(self, val):
+        if val:
+            return True
+        else:
+            return False
+            
+    def check_openai_api_key(self):
+        openai.api_key = self.cfg_private['openai']['OPENAI_API_KEY']
+        try:
+            openai.models.list()
+            return True
+        except:
+            return False
+        
+    def check_google_ocr_api_key(self):
+        # if os.path.exists(self.cfg_private['google_cloud']['path_json_file']):
+        #     os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.cfg_private['google_cloud']['path_json_file']
+        # elif os.path.exists(self.cfg_private['google_cloud']['path_json_file_service_account2']):
+        #     os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.cfg_private['google_cloud']['path_json_file_service_account2']
+        # else:
+        #     return False
+        
+        try:
+            logo_path = os.path.join(self.dir_home, 'img','logo.png')
+            client = vision.ImageAnnotatorClient()
+            with io.open(logo_path, 'rb') as image_file:
+                content = image_file.read()
+            image = vision.Image(content=content)
+            response = client.document_text_detection(image=image)
+            texts = response.text_annotations       
+            normal_cleaned_text = texts[0].description if texts else None
+            if normal_cleaned_text:
+                return True
+            else:
+                return False
+        except:
+            return False
+        
+    def check_azure_openai_api_key(self):
+        try:
+            # Initialize the Azure OpenAI client
+            model = AzureChatOpenAI(
+                deployment_name = 'gpt-35-turbo',#'gpt-35-turbo',
+                openai_api_version = self.cfg_private['openai_azure']['api_version'],
+                openai_api_key = self.cfg_private['openai_azure']['openai_api_key'],
+                azure_endpoint = self.cfg_private['openai_azure']['openai_api_base'],
+                openai_organization = self.cfg_private['openai_azure']['openai_organization'],
+            )
+            msg = HumanMessage(content="hello")
+            # self.llm_object.temperature = self.config.get('temperature')
+            response = model([msg])
+
+            # Check the response content (you might need to adjust this depending on how your AzureChatOpenAI class handles responses)
+            if response:
+                return True
+            else:
+                return False
+
+        except Exception as e:  # Use a more specific exception if possible
+            return False
+        
+    def check_mistral_api_key(self):
+        try:
+            # Initialize the Mistral Client with the API key
+            client = MistralClient(api_key=self.cfg_private['mistral']['mistral_key'])
+
+            # Create a simple message
+            messages = [ChatMessage(role="user", content="hello")]
+
+            # Send the message and get the response
+            chat_response = client.chat(
+                model="mistral-tiny",  
+                messages=messages,
+            )
+
+            # Check if the response is valid (adjust this according to the actual response structure)
+            if chat_response and chat_response.choices:
+                return True
+            else:
+                return False
+        except Exception as e:  # Replace with a more specific exception if possible
+            return False
+        
+    def check_google_vertex_genai_api_key(self):
+        results = {"palm2": False, "gemini": False}
+        try:
+            # Assuming genai and vertexai are clients for Google services
+            os.environ["GOOGLE_API_KEY"] = self.cfg_private['google_palm']['google_palm_api']
+            # genai.configure(api_key=self.cfg_private['google_palm']['google_palm_api'])
+            vertexai.init(project= self.cfg_private['google_palm']['project_id'], location=self.cfg_private['google_palm']['location'])
+
+            try:
+                model = TextGenerationModel.from_pretrained("text-bison@001")
+                response = model.predict("Hello")
+                test_response_palm = response.text
+                # llm_palm = ChatGoogleGenerativeAI(model="text-bison@001")
+                # test_response_palm = llm_palm.invoke("Hello")
+                if test_response_palm:
+                    results["palm2"] = True
+            except Exception as e:  
+                pass
+
+            try:
+                model = GenerativeModel("gemini-pro")
+                response = model.generate_content("Hello")
+                test_response_gemini = response.text
+                # llm_gemini = ChatGoogleGenerativeAI(model="gemini-pro")
+                # test_response_gemini = llm_gemini.invoke("Hello")
+                if test_response_gemini:
+                    results["gemini"] = True
+            except Exception as e:  
+                pass
+
+            return results
+        except Exception as e:  # Replace with a more specific exception if possible
+            return results
+            
+
+    def report_api_key_status(self):
+        missing_keys = []
+        present_keys = []
+
+        # Check each key and add to the respective list
+        # OpenAI key check
+        if self.has_API_key(self.cfg_private['openai']['OPENAI_API_KEY']):
+            is_valid = self.check_openai_api_key()
+            if is_valid:
+                present_keys.append('OpenAI (Valid)')
+            else:
+                present_keys.append('OpenAI (Invalid)')
+        else:
+            missing_keys.append('OpenAI')
+
+        # Azure OpenAI key check
+        if self.has_API_key(self.cfg_private['openai_azure']['api_version']):
+            is_valid = self.check_azure_openai_api_key()
+            if is_valid:
+                present_keys.append('Azure OpenAI (Valid)')
+            else:
+                present_keys.append('Azure OpenAI (Invalid)')
+        else:
+            missing_keys.append('Azure OpenAI')
+
+        # Google PALM2/Gemini key check
+        if self.has_API_key(self.cfg_private['google_palm']['google_palm_api']) and self.has_API_key(self.cfg_private['google_palm']['project_id']) and self.has_API_key(self.cfg_private['google_palm']['location']):
+            google_results = self.check_google_vertex_genai_api_key()
+            if google_results['palm2']:
+                present_keys.append('Palm2 (Valid)')
+            else:
+                present_keys.append('Palm2 (Invalid)')
+            if google_results['gemini']:
+                present_keys.append('Gemini (Valid)')
+            else:
+                present_keys.append('Gemini (Invalid)')
+        else:
+            missing_keys.append('Google VertexAI/GenAI')
+
+        # Google OCR key check
+        if self.has_API_key(self.cfg_private['google_palm']['google_palm_api']) and self.has_API_key(self.cfg_private['google_palm']['project_id']) and self.has_API_key(self.cfg_private['google_palm']['location']):
+            is_valid = self.check_google_ocr_api_key()
+            if is_valid:
+                present_keys.append('Google OCR (Valid)')
+            else:
+                present_keys.append('Google OCR (Invalid)')
+        else:
+            missing_keys.append('Google OCR')
+
+        # Mistral key check
+        if self.has_API_key(self.cfg_private['mistral']['mistral_key']):
+            is_valid = self.check_mistral_api_key()
+            if is_valid:
+                present_keys.append('Mistral (Valid)')
+            else:
+                present_keys.append('Mistral (Invalid)')
+        else:
+            missing_keys.append('Mistral')
+
+
+        if self.has_API_key(self.cfg_private['here']['api_key']):
+            present_keys.append('HERE Geocode (Valid)')
+        else:
+            missing_keys.append('HERE Geocode (Invalid)')
+
+        if self.has_API_key(self.cfg_private['open_cage_geocode']['api_key']):
+            present_keys.append('OpenCage Geocode (Valid)')
+        else:
+            missing_keys.append('OpenCage Geocode (Invalid)')
+
+        # Create a report string
+        report = "API Key Status Report:\n"
+        report += "Present Keys: " + ", ".join(present_keys) + "\n"
+        report += "Missing Keys: " + ", ".join(missing_keys) + "\n"
+
+        # print(report)
+        return present_keys, missing_keys, self.formatted_date

vouchervision/DEP_prompt_catalog.py

@@ -0,0 +1,1322 @@
+from dataclasses import dataclass
+from langchain_core.pydantic_v1 import BaseModel, Field, create_model
+import yaml, json
+
+
+#  catalog = PromptCatalog(OCR="Sample OCR text", domain_knowledge_example="Sample domain knowledge", similarity="0.9")
+
+
+### Required if you want to use the Pydantic JSON parser for langchain
+class SLTPvA(BaseModel):
+    catalogNumber: str = Field(description="Barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits.")
+    order: str = Field(description="The full scientific name of the order in which the taxon is classified. Order must be capitalized.")
+    family: str = Field(description="The full scientific name of the family in which the taxon is classified. Family must be capitalized.")
+    scientificName: str = Field(description="The scientific name of the taxon including genus, specific epithet, and any lower classifications.")
+    scientificNameAuthorship: str = Field(description="The authorship information for the scientificName formatted according to the conventions of the applicable Darwin Core nomenclaturalCode.")
+    genus: str = Field(description="Taxonomic determination to genus. Genus must be capitalized. If genus is not present use the taxonomic family name followed by the word 'indet'.")
+    subgenus: str = Field(description="The full scientific name of the subgenus in which the taxon is classified. Values should include the genus to avoid homonym confusion.")
+    specificEpithet: str = Field(description="The name of the first or species epithet of the scientificName. Only include the species epithet.")
+    infraspecificEpithet: str = Field(description="The name of the lowest or terminal infraspecific epithet of the scientificName, excluding any rank designation.")
+    identifiedBy: str = Field(description="A comma separated list of names of people, groups, or organizations who assigned the taxon to the subject organism. This is not the specimen collector.")
+    recordedBy: str = Field(description="A comma separated list of names of people, groups, or organizations responsible for observing, recording, collecting, or presenting the original specimen. The primary collector or observer should be listed first.")
+    recordNumber: str = Field(description="An identifier given to the occurrence at the time it was recorded. Often serves as a link between str = Field notes and an occurrence record, such as a specimen collector's number.")
+    verbatimEventDate: str = Field(description="The verbatim original representation of the date and time information for when the specimen was collected. Date of collection exactly as it appears on the label. Do not change the format or correct typos.")
+    eventDate:str = Field(description=" Date the specimen was collected formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples '0000-00-00' if the entire date is unknown, 'YYYY-00-00' if only the year is known, and 'YYYY-MM-00' if year and month are known but day is not.")
+    habitat: str = Field(description="A category or description of the habitat in which the specimen collection event occurred.")
+    occurrenceRemarks: str = Field(description="Text describing the specimen's geographic location. Text describing the appearance of the specimen. A statement about the presence or absence of a taxon at a the collection location. Text describing the significance of the specimen, such as a specific expedition or notable collection. Description of plant features such as leaf shape, size, color, stem texture, height, flower structure, scent, fruit or seed characteristics, root system type, overall growth habit and form, any notable aroma or secretions, presence of hairs or bristles, and any other distinguishing morphological or physiological characteristics.")
+    country: str = Field(description="The name of the country or major administrative unit in which the specimen was originally collected.")
+    stateProvince: str = Field(description="The name of the next smaller administrative region than country (state, province, canton, department, region, etc.) in which the specimen was originally collected.")
+    county:str = Field(description=" The full, unabbreviated name of the next smaller administrative region than stateProvince (county, shire, department, parish etc.) in which the specimen was originally collected.")
+    municipality: str = Field(description="The full, unabbreviated name of the next smaller administrative region than county (city, municipality, etc.) in which the specimen was originally collected.")
+    locality: str = Field(description="Description of geographic location, landscape, landmarks, regional features, nearby places, or any contextual information aiding in pinpointing the exact origin or location of the specimen.")
+    degreeOfEstablishment: str = Field(description="Cultivated plants are intentionally grown by humans. In text descriptions, look for planting dates, garden locations, ornamental, cultivar names, garden, or farm to indicate cultivated plant. Other possible designiations include - unknown, native, captive, cultivated, released, failing, casual, reproducing, established, colonising, invasive, widespreadInvasive. Based on the OCR text, assign the most suitable designiation and use the term unknown as the default designiation.")
+    decimalLatitude: str = Field(description="Latitude decimal coordinate. Correct and convert the verbatim location coordinates to conform with the decimal degrees GPS coordinate format.")
+    decimalLongitude: str = Field(description="Longitude decimal coordinate. Correct and convert the verbatim location coordinates to conform with the decimal degrees GPS coordinate format.")
+    verbatimCoordinates: str = Field(description="Verbatim location coordinates as they appear on the label. Do not convert formats. Possible coordinate types include [Lat, Long, UTM, TRS].")
+    minimumElevationInMeters: str = Field(description="Minimum elevation or altitude in meters. Only if units are explicit then convert from feet ('ft' or 'ft.'' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer.")
+    maximumElevationInMeters: str = Field(description="Maximum elevation or altitude in meters. If only one elevation is present, then max_elevation should be set to the null_value. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer.")
+   
+
+
+
+@dataclass
+class PromptCatalog:
+    domain_knowledge_example: str = ""
+    similarity: str = ""
+    OCR: str = ""
+    n_fields: int = 0
+
+    # def PROMPT_UMICH_skeleton_all_asia(self, OCR=None, domain_knowledge_example=None, similarity=None):
+    def prompt_v1_verbose(self, OCR=None, domain_knowledge_example=None, similarity=None):
+        self.OCR = OCR or self.OCR
+        self.domain_knowledge_example = domain_knowledge_example or self.domain_knowledge_example
+        self.similarity = similarity or self.similarity
+        self.n_fields = 22 or self.n_fields
+
+        set_rules = """
+        Please note that your task is to generate a dictionary, following the below rules:
+        1. Refactor the unstructured OCR text into a dictionary based on the reference dictionary structure (ref_dict).
+        2. Each field of OCR corresponds to a column of the ref_dict. You should correctly map the values from OCR to the respective fields in ref_dict.
+        3. If the OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None".
+        4. If there is a field in the ref_dict that does not have a corresponding value in the OCR text, fill it based on your knowledge but don't generate new information.
+        5. Do not use any text from the ref_dict values in the new dict, but you must use the headers from ref_dict.
+        6. Duplicate dictionary fields are not allowed.
+        7. Only return the new dictionary. You should not explain your answer.
+        8. Your output should be a Python dictionary represented as a JSON string.
+        """
+
+        umich_all_asia_rules = """{
+            "Catalog Number": {
+                "format": "[Catalog Number]",
+                "null_value": "",
+                "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits"
+            },
+            "Genus": {
+                "format": "[Genus] or '[Family] indet' if no genus",
+                "null_value": "",
+                "description": "Taxonomic determination to genus, do capitalize genus"
+            },
+            "Species": {
+                "format": "[species] or 'indet' if no species",
+                "null_value": "",
+                "description": "Taxonomic determination to species, do not capitalize species"
+            },
+            "subspecies": {
+                "format": "[subspecies]",
+                "null_value": "",
+                "description": "Taxonomic determination to subspecies (subsp.)"
+            },
+            "variety": {
+                "format": "[variety]",
+                "null_value": "",
+                "description": "Taxonomic determination to variety (var)"
+            },
+            "forma": {
+                "format": "[form]",
+                "null_value": "",
+                "description": "Taxonomic determination to form (f.)"
+            },
+            "Country": {
+                "format": "[Country]",
+                "null_value": "",
+                "description": "Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abbreviation is given"
+            },
+            "State": {
+                "format": "[Adm. Division 1]",
+                "null_value": "",
+                "description": "Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word"
+            },
+            "County": {
+                "format": "[Adm. Division 2]",
+                "null_value": "",
+                "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word"
+            },
+            "Locality Name": {
+                "format": "verbatim, if no geographic info: 'no data provided on label of catalog no: [######]', or if illegible: 'locality present but illegible/not translated for catalog no: #######', or if no named locality: 'no named locality for catalog no: #######'",
+                "description": "Description of geographic location or landscape"
+            },
+            "Min Elevation": {
+                "format": "elevation integer",
+                "null_value": "",
+                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given"
+            },
+            "Max Elevation": {
+                "format": "elevation integer",
+                "null_value": "",
+                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise"
+            },
+            "Elevation Units": {
+                "format": "m",
+                "null_value": "",
+                "description": "'m' only if an elevation is present"
+            },
+            "Verbatim Coordinates": {
+                "format": "[Lat, Long | UTM | TRS]",
+                "null_value": "",
+                "description": "Verbatim coordinates as they appear on the label, fix typos to match standardized GPS coordinate format"
+            },
+            "Datum": {
+                "format": "[WGS84, NAD23 etc.]",
+                "null_value": "",
+                "description": "GPS Datum of coordinates on label; empty string "" if GPS coordinates are not in OCR"
+            },
+            "Cultivated": {
+                "format": "yes",
+                "null_value": "",
+                "description": "Indicates if specimen was grown in cultivation"
+            },
+            "Habitat": {
+                "format": "verbatim",
+                "null_value": "",
+                "description": "Description of habitat or location where specimen was collected, ignore descriptions of the plant itself"
+            },
+            "Collectors": {
+                "format": "[Collector]",
+                "null_value": "not present",
+                "description": "Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas"
+            },
+            "Collector Number": {
+                "format": "[Collector No.]",
+                "null_value": "s.n.",
+                "description": "Sequential number assigned to collection, associated with the collector"
+            },
+            "Verbatim Date": {
+                "format": "verbatim",
+                "null_value": "s.d.",
+                "description": "Date of collection exactly as it appears on the label"
+            },
+            "Date": {
+                "format": "[yyyy-mm-dd]",
+                "null_value": "",
+                "description": "Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e., 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day"
+            },
+            "End Date": {
+                "format": "[yyyy-mm-dd]",
+                "null_value": "",
+                "description": "If date range is listed, later date of collection range"
+            }
+        }"""
+
+        structure = """{"Dictionary":
+                            {
+                            "Catalog Number": [Catalog Number],
+                            "Genus": [Genus],
+                            "Species": [species],
+                            "subspecies": [subspecies],
+                            "variety": [variety],
+                            "forma": [forma],
+                            "Country": [Country],
+                            "State": [State],
+                            "County": [County],
+                            "Locality Name": [Locality Name],
+                            "Min Elevation": [Min Elevation],
+                            "Max Elevation": [Max Elevation],
+                            "Elevation Units": [Elevation Units],
+                            "Verbatim Coordinates": [Verbatim Coordinates],
+                            "Datum": [Datum],
+                            "Cultivated": [Cultivated],
+                            "Habitat": [Habitat],
+                            "Collectors": [Collectors],
+                            "Collector Number": [Collector Number],
+                            "Verbatim Date": [Verbatim Date],
+                            "Date": [Date],
+                            "End Date": [End Date]
+                            },
+                        "SpeciesName": {"taxonomy": [Genus_species]}}"""
+
+        prompt = f"""I'm providing you with a set of rules, an unstructured OCR text, and a reference dictionary (domain knowledge). Your task is to convert the OCR text into a structured dictionary that matches the structure of the reference dictionary. Please follow the rules strictly.
+        The rules are as follows:
+        {set_rules}
+        The unstructured OCR text is:
+        {self.OCR}
+        The reference dictionary, which provides an example of the output structure and has an embedding distance of {self.similarity} to the OCR, is:
+        {self.domain_knowledge_example}
+        Some dictionary fields have special requirements. These requirements specify the format for each field, and are given below:
+        {umich_all_asia_rules}
+        Please refactor the OCR text into a dictionary, following the rules and the reference structure:
+        {structure}
+        """
+
+        xlsx_headers = ["Catalog Number","Genus","Species","subspecies","variety","forma","Country","State","County","Locality Name","Min Elevation","Max Elevation","Elevation Units","Verbatim Coordinates","Datum","Cultivated","Habitat","Collectors","Collector Number","Verbatim Date","Date","End Date"]
+
+
+        return prompt, self.n_fields, xlsx_headers
+    
+    def prompt_v1_verbose_noDomainKnowledge(self, OCR=None):
+        self.OCR = OCR or self.OCR
+        self.n_fields = 22 or self.n_fields
+
+        set_rules = """
+        Please note that your task is to generate a dictionary, following the below rules:
+        1. Refactor the unstructured OCR text into a dictionary based on the reference dictionary structure (ref_dict).
+        2. Each field of OCR corresponds to a column of the ref_dict. You should correctly map the values from OCR to the respective fields in ref_dict.
+        3. If the OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None".
+        4. If there is a field in the ref_dict that does not have a corresponding value in the OCR text, fill it based on your knowledge but don't generate new information.
+        5. Do not use any text from the ref_dict values in the new dict, but you must use the headers from ref_dict.
+        6. Duplicate dictionary fields are not allowed.
+        7. Only return the new dictionary. You should not explain your answer.
+        8. Your output should be a Python dictionary represented as a JSON string.
+        """
+
+        umich_all_asia_rules = """{
+            "Catalog Number": {
+                "format": "[Catalog Number]",
+                "null_value": "",
+                "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits"
+            },
+            "Genus": {
+                "format": "[Genus] or '[Family] indet' if no genus",
+                "null_value": "",
+                "description": "Taxonomic determination to genus, do capitalize genus"
+            },
+            "Species": {
+                "format": "[species] or 'indet' if no species",
+                "null_value": "",
+                "description": "Taxonomic determination to species, do not capitalize species"
+            },
+            "subspecies": {
+                "format": "[subspecies]",
+                "null_value": "",
+                "description": "Taxonomic determination to subspecies (subsp.)"
+            },
+            "variety": {
+                "format": "[variety]",
+                "null_value": "",
+                "description": "Taxonomic determination to variety (var)"
+            },
+            "forma": {
+                "format": "[form]",
+                "null_value": "",
+                "description": "Taxonomic determination to form (f.)"
+            },
+            "Country": {
+                "format": "[Country]",
+                "null_value": "",
+                "description": "Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abbreviation is given"
+            },
+            "State": {
+                "format": "[Adm. Division 1]",
+                "null_value": "",
+                "description": "Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word"
+            },
+            "County": {
+                "format": "[Adm. Division 2]",
+                "null_value": "",
+                "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word"
+            },
+            "Locality Name": {
+                "format": "verbatim, if no geographic info: 'no data provided on label of catalog no: [######]', or if illegible: 'locality present but illegible/not translated for catalog no: #######', or if no named locality: 'no named locality for catalog no: #######'",
+                "description": "Description of geographic location or landscape"
+            },
+            "Min Elevation": {
+                "format": "elevation integer",
+                "null_value": "",
+                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given"
+            },
+            "Max Elevation": {
+                "format": "elevation integer",
+                "null_value": "",
+                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise"
+            },
+            "Elevation Units": {
+                "format": "m",
+                "null_value": "",
+                "description": "'m' only if an elevation is present"
+            },
+            "Verbatim Coordinates": {
+                "format": "[Lat, Long | UTM | TRS]",
+                "null_value": "",
+                "description": "Verbatim coordinates as they appear on the label, fix typos to match standardized GPS coordinate format"
+            },
+            "Datum": {
+                "format": "[WGS84, NAD23 etc.]",
+                "null_value": "",
+                "description": "GPS Datum of coordinates on label; empty string "" if GPS coordinates are not in OCR"
+            },
+            "Cultivated": {
+                "format": "yes",
+                "null_value": "",
+                "description": "Indicates if specimen was grown in cultivation"
+            },
+            "Habitat": {
+                "format": "verbatim",
+                "null_value": "",
+                "description": "Description of habitat or location where specimen was collected, ignore descriptions of the plant itself"
+            },
+            "Collectors": {
+                "format": "[Collector]",
+                "null_value": "not present",
+                "description": "Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas"
+            },
+            "Collector Number": {
+                "format": "[Collector No.]",
+                "null_value": "s.n.",
+                "description": "Sequential number assigned to collection, associated with the collector"
+            },
+            "Verbatim Date": {
+                "format": "verbatim",
+                "null_value": "s.d.",
+                "description": "Date of collection exactly as it appears on the label"
+            },
+            "Date": {
+                "format": "[yyyy-mm-dd]",
+                "null_value": "",
+                "description": "Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e., 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day"
+            },
+            "End Date": {
+                "format": "[yyyy-mm-dd]",
+                "null_value": "",
+                "description": "If date range is listed, later date of collection range"
+            }
+        }"""
+
+        structure = """{"Dictionary":
+                            {
+                            "Catalog Number": [Catalog Number],
+                            "Genus": [Genus],
+                            "Species": [species],
+                            "subspecies": [subspecies],
+                            "variety": [variety],
+                            "forma": [forma],
+                            "Country": [Country],
+                            "State": [State],
+                            "County": [County],
+                            "Locality Name": [Locality Name],
+                            "Min Elevation": [Min Elevation],
+                            "Max Elevation": [Max Elevation],
+                            "Elevation Units": [Elevation Units],
+                            "Verbatim Coordinates": [Verbatim Coordinates],
+                            "Datum": [Datum],
+                            "Cultivated": [Cultivated],
+                            "Habitat": [Habitat],
+                            "Collectors": [Collectors],
+                            "Collector Number": [Collector Number],
+                            "Verbatim Date": [Verbatim Date],
+                            "Date": [Date],
+                            "End Date": [End Date]
+                            },
+                        "SpeciesName": {"taxonomy": [Genus_species]}}"""
+
+        prompt = f"""I'm providing you with a set of rules, an unstructured OCR text, and a reference dictionary (domain knowledge). Your task is to convert the OCR text into a structured dictionary that matches the structure of the reference dictionary. Please follow the rules strictly.
+        The rules are as follows:
+        {set_rules}
+        The unstructured OCR text is:
+        {self.OCR}
+        Some dictionary fields have special requirements. These requirements specify the format for each field, and are given below:
+        {umich_all_asia_rules}
+        Please refactor the OCR text into a dictionary, following the rules and the reference structure:
+        {structure}
+        """
+
+        xlsx_headers = ["Catalog Number","Genus","Species","subspecies","variety","forma","Country","State","County","Locality Name","Min Elevation","Max Elevation","Elevation Units","Verbatim Coordinates","Datum","Cultivated","Habitat","Collectors","Collector Number","Verbatim Date","Date","End Date"]
+
+        return prompt, self.n_fields, xlsx_headers
+    
+    def prompt_v2_json_rules(self, OCR=None):
+        self.OCR = OCR or self.OCR
+        self.n_fields = 26 or self.n_fields
+
+        set_rules = """
+        1. Refactor the unstructured OCR text into a dictionary based on the JSON structure outlined below.
+        2. You should map the unstructured OCR text to the appropriate JSON key and then populate the field based on its rules.
+        3. Some JSON key fields are permitted to remain empty if the corresponding information is not found in the unstructured OCR text.
+        4. Ignore any information in the OCR text that doesn't fit into the defined JSON structure.
+        5. Duplicate dictionary fields are not allowed.
+        6. Ensure that all JSON keys are in lowercase.
+        7. Ensure that new JSON field values follow sentence case capitalization.
+        7. Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+        8. Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+        9. Only return a JSON dictionary represented as a string. You should not explain your answer.
+        """
+
+        dictionary_field_format_descriptions = """
+        The next section of instructions outlines how to format the JSON dictionary. The keys are the same as those of the final formatted JSON object.
+        For each key there is a format requirement that specifies how to transcribe the information for that key. 
+        The possible formatting options are:
+        1. "verbatim transcription" - field is populated with verbatim text from the unformatted OCR.
+        2. "spell check transcription" - field is populated with spelling corrected text from the unformatted OCR.
+        3. "boolean yes no" - field is populated with only yes or no.
+        4. "integer" - field is populated with only an integer.
+        5. "[list]" - field is populated from one of the values in the list.
+        6. "yyyy-mm-dd" - field is populated with a date in the format year-month-day.
+        The desired null value is also given. Populate the field with the null value of the information for that key is not present in the unformatted OCR text. 
+        """
+
+        json_template_rules = """
+            {"Dictionary":{
+                "catalog_number": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits."
+                },
+                "genus": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to genus. Genus must be capitalized. If genus is not present use the taxonomic family name followed by the word 'indet'."
+                },
+                "species": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to species, do not capitalize species."
+                },
+                "subspecies": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to subspecies (subsp.)."
+                },
+                "variety": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to variety (var)."
+                },
+                "forma": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to form (f.)."
+                },
+                "country": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Country that corresponds to the current geographic location of collection. Capitalize first letter of each word. If abbreviation is given populate field with the full spelling of the country's name."
+                },
+                "state": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Administrative division 1 that corresponds to the current geographic location of collection. Capitalize first letter of each word. Administrative division 1 is equivalent to a U.S. State."
+                },
+                "county": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word. Administrative division 2 is equivalent to a U.S. county, parish, borough."
+                },
+                "locality_name": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Description of geographic location, landscape, landmarks, regional features, nearby places, or any contextual information aiding in pinpointing the exact origin or site of the specimen."
+                },
+                "min_elevation": {
+                    "format": "integer",
+                    "null_value": "",
+                    "description": "Minimum elevation or altitude in meters. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
+                },
+                "max_elevation": {
+                    "format": "integer",
+                    "null_value": "",
+                    "description": "Maximum elevation or altitude in meters. If only one elevation is present, then max_elevation should be set to the null_value. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
+                },
+                "elevation_units": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Elevation units must be meters. If min_elevation field is populated, then elevation_units: 'm'. Otherwise elevation_units: ''."
+                },
+                "verbatim_coordinates": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Verbatim location coordinates as they appear on the label. Do not convert formats. Possible coordinate types are one of [Lat, Long, UTM, TRS]."
+                },
+                "decimal_coordinates": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Correct and convert the verbatim location coordinates to conform with the decimal degrees GPS coordinate format."
+                },
+                "datum": {
+                    "format": "[WGS84, WGS72, WGS66, WGS60, NAD83, NAD27, OSGB36, ETRS89, ED50, GDA94, JGD2011, Tokyo97, KGD2002, TWD67, TWD97, BJS54, XAS80, GCJ-02, BD-09, PZ-90.11, GTRF, CGCS2000, ITRF88, ITRF89, ITRF90, ITRF91, ITRF92, ITRF93, ITRF94, ITRF96, ITRF97, ITRF2000, ITRF2005, ITRF2008, ITRF2014, Hong Kong Principal Datum, SAD69]",
+                    "null_value": "",
+                    "description": "Datum of location coordinates. Possible values are include in the format list. Leave field blank if unclear."
+                },
+                "cultivated": {
+                    "format": "boolean yes no",
+                    "null_value": "",
+                    "description": "Cultivated plants are intentionally grown by humans. In text descriptions, look for planting dates, garden locations, ornamental, cultivar names, garden, or farm to indicate cultivated plant."
+                },
+                "habitat": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Description of a plant's habitat or the location where the specimen was collected. Ignore descriptions of the plant itself."
+                },
+                "plant_description": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Description of plant features such as leaf shape, size, color, stem texture, height, flower structure, scent, fruit or seed characteristics, root system type, overall growth habit and form, any notable aroma or secretions, presence of hairs or bristles, and any other distinguishing morphological or physiological characteristics."
+                },
+                "collectors": {
+                    "format": "verbatim transcription",
+                    "null_value": "not present",
+                    "description": "Full name(s) of the individual(s) responsible for collecting the specimen. When multiple collectors are involved, their names should be separated by commas."
+                },
+                "collector_number": {
+                    "format": "verbatim transcription",
+                    "null_value": "s.n.",
+                    "description": "Unique identifier or number that denotes the specific collecting event and associated with the collector."
+                },
+                "determined_by": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Full name of the individual responsible for determining the taxanomic name of the specimen. Sometimes the name will be near to the characters 'det' to denote determination. This name may be isolated from other names in the unformatted OCR text."
+                },
+                "multiple_names": {
+                    "format": "boolean yes no",
+                    "null_value": "",
+                    "description": "Indicate whether multiple people or collector names are present in the unformatted OCR text. If you see more than one person's name the value is 'yes'; otherwise the value is 'no'."
+                },
+                "verbatim_date": {
+                    "format": "verbatim transcription",
+                    "null_value": "s.d.",
+                    "description": "Date of collection exactly as it appears on the label. Do not change the format or correct typos."
+                },
+                "date": {
+                    "format": "yyyy-mm-dd",
+                    "null_value": "",
+                    "description": "Date the specimen was collected formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire date is unknown, 'YYYY-00-00' if only the year is known, and 'YYYY-MM-00' if year and month are known but day is not."
+                },
+                "end_date": {
+                    "format": "yyyy-mm-dd",
+                    "null_value": "",
+                    "description": "If a date range is provided, this represents the later or ending date of the collection period, formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire end date is unknown, 'YYYY-00-00' if only the year of the end date is known, and 'YYYY-MM-00' if year and month of the end date are known but the day is not."
+                },
+            },
+            "SpeciesName": {
+                "taxonomy": [Genus_species]}
+            }"""
+        
+        structure = """{"Dictionary":
+                            {
+                            "catalog_number": "",
+                            "genus": "",
+                            "species": "",
+                            "subspecies": "",
+                            "variety": "",
+                            "forma": "",
+                            "country": "",
+                            "state": "",
+                            "county": "",
+                            "locality_name": "",
+                            "min_elevation": "",
+                            "max_elevation": "",
+                            "elevation_units": "",
+                            "verbatim_coordinates": "",
+                            "decimal_coordinates": "",
+                            "datum": "",
+                            "cultivated": "",
+                            "habitat": "",
+                            "plant_description": "",
+                            "collectors": "",
+                            "collector_number": "",
+                            "determined_by": "",
+                            "multiple_names": "",
+                            "verbatim_date":"" ,
+                            "date": "",
+                            "end_date": ""
+                            },
+                        "SpeciesName": {"taxonomy": ""}}"""
+
+        prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
+        The rules are:
+        {set_rules}
+        The unstructured OCR text is:
+        {self.OCR}
+        {dictionary_field_format_descriptions}
+        This is the JSON template that includes instructions for each key:
+        {json_template_rules}
+        Please populate the following JSON dictionary based on the rules and the unformatted OCR text:
+        {structure}
+        """
+
+        xlsx_headers = ["catalog_number","genus","species","subspecies","variety","forma","country","state","county","locality_name","min_elevation","max_elevation","elevation_units","verbatim_coordinates","decimal_coordinates","datum","cultivated","habitat","plant_description","collectors","collector_number","determined_by","multiple_names","verbatim_date","date","end_date"]
+        
+        return prompt, self.n_fields, xlsx_headers
+    
+    #############################################################################################
+    #############################################################################################
+    #############################################################################################
+    #############################################################################################
+    # These are for dynamically creating your own prompts with n-columns
+
+
+    def prompt_SLTP(self, rules_config_path, OCR=None, is_palm=False):
+        self.OCR = OCR
+
+        self.rules_config_path = rules_config_path
+        self.rules_config = self.load_rules_config()
+
+        self.instructions = self.rules_config['instructions']
+        self.json_formatting_instructions = self.rules_config['json_formatting_instructions']
+
+        self.rules_list = self.rules_config['rules']
+        self.n_fields = len(self.rules_config['rules'])
+
+        # Set the rules for processing OCR into JSON format
+        self.rules = self.create_rules(is_palm)
+
+        self.structure, self.dictionary_structure = self.create_structure(is_palm)
+
+        ''' between  instructions and json_formatting_instructions. Made the prompt too long. Better performance without it
+        The unstructured OCR text is:
+        {self.OCR}
+        '''
+        if is_palm:
+            prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
+                The rules are:
+                {self.instructions}
+                {self.json_formatting_instructions}
+                This is the JSON template that includes instructions for each key:
+                {self.rules}
+                The unstructured OCR text is:
+                {self.OCR}
+                Please populate the following JSON dictionary based on the rules and the unformatted OCR text:
+                {self.structure}
+                {self.structure}
+                {self.structure}
+                """
+        else:
+            prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
+                The rules are:
+                {self.instructions}
+                {self.json_formatting_instructions}
+                This is the JSON template that includes instructions for each key:
+                {self.rules}
+                The unstructured OCR text is:
+                {self.OCR}
+                Please populate the following JSON dictionary based on the rules and the unformatted OCR text:
+                {self.structure}
+                """
+        # xlsx_headers = self.generate_xlsx_headers(is_palm)
+        
+        # return prompt, self.PromptJSONModel, self.n_fields, xlsx_headers
+        return prompt, self.dictionary_structure
+
+    def load_rules_config(self):
+        with open(self.rules_config_path, 'r') as stream:
+            try:
+                return yaml.safe_load(stream)
+            except yaml.YAMLError as exc:
+                print(exc)
+                return None
+
+    def create_rules(self, is_palm=False):
+        dictionary_structure = {key: value for key, value in self.rules_list.items()}
+
+        # Convert the structure to a JSON string without indentation
+        structure_json_str = json.dumps(dictionary_structure, sort_keys=False)
+        return structure_json_str
+    
+    def create_structure(self, is_palm=False):
+        # Create fields for the Pydantic model dynamically
+        fields = {key: (str, Field(default=value, description=value)) for key, value in self.rules_list.items()}
+
+        # Dynamically create the Pydantic model
+        DynamicJSONParsingModel = create_model('SLTPvA', **fields)
+        DynamicJSONParsingModel_use = DynamicJSONParsingModel()
+
+        # Define the structure for the "Dictionary" section
+        dictionary_fields = {key: (str, Field(default='', description="")) for key in self.rules_list.keys()}
+        
+        # Dynamically create the "Dictionary" Pydantic model
+        PromptJSONModel = create_model('PromptJSONModel', **dictionary_fields)
+
+        # Convert the model to JSON string (for demonstration)
+        dictionary_structure = PromptJSONModel().dict()
+        structure_json_str = json.dumps(dictionary_structure, sort_keys=False, indent=4)
+        return structure_json_str, dictionary_structure
+
+
+    def generate_xlsx_headers(self, is_palm):
+        # Extract headers from the 'Dictionary' keys in the JSON template rules
+        if is_palm:
+            xlsx_headers = list(self.rules_list.keys())
+            return xlsx_headers     
+        else:
+            xlsx_headers = list(self.rules_list.keys())
+            return xlsx_headers
+    
+    def prompt_v2_custom_redo(self, incorrect_json, is_palm):
+        # Load the existing rules and structure
+        self.rules_config = self.load_rules_config()
+        # self.rules = self.create_rules(is_palm)
+        self.structure, self.dictionary_structure = self.create_structure(is_palm)
+        
+        # Generate the prompt using the loaded rules and structure
+        if is_palm:
+            prompt = f"""The incorrectly formatted JSON dictionary below is not valid. It contains an error that prevents it from loading with the Python command json.loads().
+                The incorrectly formatted JSON dictionary below is the literal string returned by a previous function and the error may be caused by markdown formatting.
+                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
+                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+                The incorrectly formatted JSON dictionary: {incorrect_json}
+                The output JSON structure: {self.structure}
+                The output JSON structure: {self.structure}
+                The output JSON structure: {self.structure}
+                Please reformat the incorrectly formatted JSON dictionary given the output JSON structure: """
+        else:
+            prompt = f"""The incorrectly formatted JSON dictionary below is not valid. It contains an error that prevents it from loading with the Python command json.loads().
+                The incorrectly formatted JSON dictionary below is the literal string returned by a previous function and the error may be caused by markdown formatting.
+                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
+                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+                The incorrectly formatted JSON dictionary: {incorrect_json}
+                The output JSON structure: {self.structure}
+                Please reformat the incorrectly formatted JSON dictionary given the output JSON structure: """
+        return prompt
+
+    #############################################################################################
+    #############################################################################################
+    #############################################################################################
+    #############################################################################################
+    def prompt_gpt_redo_v1(self, incorrect_json):       
+        structure = """Below is the correct JSON formatting. Modify the text to conform to the following format, fixing the incorrect JSON:
+        {"Dictionary":
+            {
+            "Catalog Number": [Catalog Number],
+            "Genus": [Genus],
+            "Species": [species],
+            "subspecies": [subspecies],
+            "variety": [variety],
+            "forma": [forma],
+            "Country": [Country],
+            "State": [State],
+            "County": [County],
+            "Locality Name": [Locality Name],
+            "Min Elevation": [Min Elevation],
+            "Max Elevation": [Max Elevation],
+            "Elevation Units": [Elevation Units],
+            "Verbatim Coordinates": [Verbatim Coordinates],
+            "Datum": [Datum],
+            "Cultivated": [Cultivated],
+            "Habitat": [Habitat],
+            "Collectors": [Collectors],
+            "Collector Number": [Collector Number],
+            "Verbatim Date": [Verbatim Date],
+            "Date": [Date],
+            "End Date": [End Date]
+            },
+        "SpeciesName": {"taxonomy": [Genus_species]}}"""
+
+        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
+                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
+                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+                The incorrectly formatted JSON dictionary: {incorrect_json}
+                The output JSON structure: {structure}
+                The refactored JSON disctionary: """
+        return prompt
+    
+    def prompt_gpt_redo_v2(self, incorrect_json):
+        structure = """
+            {"Dictionary":{
+                "catalog_number": "",
+                "genus": "",
+                "species": "".
+                "subspecies": "",
+                "variety": "",
+                "forma":"",
+                "country": "",
+                "state": "",
+                "county": "",
+                "locality_name": "",
+                "min_elevation": "",
+                "max_elevation": "",
+                "elevation_units": "',
+                "verbatim_coordinates": "",
+                "decimal_coordinates": "",
+                "datum": "",
+                "cultivated": "",
+                "habitat": "",
+                "plant_description": "",
+                "collectors": "",
+                "collector_number": "",
+                "determined_by": "",
+                "multiple_names": "',
+                "verbatim_date": "",
+                "date": "",
+                "end_date": "",
+                },
+            "SpeciesName": {"taxonomy": [Genus_species]}}"""
+        
+        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
+                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
+                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+                The incorrectly formatted JSON dictionary: {incorrect_json}
+                The output JSON structure: {structure}
+                The refactored JSON disctionary: """
+        return prompt
+    #####################################################################################################################################
+    #####################################################################################################################################
+    def prompt_v1_palm2(self, in_list, out_list, OCR=None):
+        self.OCR = OCR or self.OCR
+        set_rules = """1. Your job is to return a new dict based on the structure of the reference dict ref_dict and these are your rules. 
+                        2. You must look at ref_dict and refactor the new text called OCR to match the same formatting. 
+                        3. OCR contains unstructured text inside of [], use your knowledge to put the OCR text into the correct ref_dict column. 
+                        4. If OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None" and skip all other steps.
+                        5. If there is a field that does not have a direct proxy in the OCR text, you can fill it in based on your knowledge, but you cannot generate new information.
+                        6. Never put text from the ref_dict values into the new dict, but you must use the headers from ref_dict. 
+                        7. There cannot be duplicate dictionary fields.
+                        8. Only return the new dict, do not explain your answer.
+                        9. Do not include quotation marks in content, only use quotation marks to represent values in dictionaries.
+                        10. For GPS coordinates only use Decimal Degrees (D.D°)
+                        11. "Given the input text, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values."""
+
+        umich_all_asia_rules = """
+        "Catalog Number" - {"format": "[barcode]", "null_value": "", "description": the barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits}
+        "Genus" - {"format": "[Genus]" or "[Family] indet" if no genus", "null_value": "", "description": taxonomic determination to genus, do captalize genus}
+        "Species"- {"format": "[species]" or "indet" if no species, "null_value": "", "description": taxonomic determination to species, do not captalize species}
+        "subspecies" - {"format": "[subspecies]", "null_value": "", "description": taxonomic determination to subspecies (subsp.)}
+        "variety" - {"format": "[variety]", "null_value": "", "description": taxonomic determination to variety (var)}
+        "forma" - {"format": "[form]", "null_value": "", "description": taxonomic determination to form (f.)}
+
+        "Country" - {"format": "[Country]", "null_value": "no data", "description": Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abreviation is given}
+        "State" - {"format": "[Adm. Division 1]", "null_value": "no data", "description": Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word}
+        "County" - {"format": "[Adm. Division 2]", "null_value": "no data", "description": Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word}
+        "Locality Name" - {"format": "verbatim", if no geographic info: "no data provided on label of catalog no: [######]", or if illegible: "locality present but illegible/not translated for catalog no: #######", or if no named locality: "no named locality for catalog no: #######", "description": "Description of geographic location or landscape"}
+
+        "Min Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given}
+        "Max Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise}
+        "Elevation Units" - {format: "m", "null_value": "","description": "m" only if an elevation is present}
+        
+        "Verbatim Coordinates" - {"format": "[Lat, Long | UTM | TRS]", "null_value": "", "description": Convert coordinates to Decimal Degrees (D.D°) format, do not use Minutes, Seconds or quotation marks}
+
+        "Datum" - {"format": "[WGS84, NAD23 etc.]", "null_value": "not present", "description": Datum of coordinates on label; "" is GPS coordinates are not in OCR}
+        "Cultivated" - {"format": "yes", "null_value": "", "description": Indicates if specimen was grown in cultivation}
+        "Habitat" - {"format": "verbatim", "null_value": "", "description": Description of habitat or location where specimen was collected, ignore descriptions of the plant itself}
+        "Collectors" - {"format": "[Collector]", "null_value": "not present", "description": Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas}
+        "Collector Number" - {"format": "[Collector No.]", "null_value": "s.n.", "description": Sequential number assigned to collection, associated with the collector}
+        "Verbatim Date" - {"format": "verbatim", "null_value": "s.d.", "description": Date of collection exactly as it appears on the label}
+        "Date" - {"format": "[yyyy-mm-dd]", "null_value": "", "description": Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e. 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day}
+        "End Date" - {"format": "[yyyy-mm-dd]", "null_value": "", "description": If date range is listed, later date of collection range}
+        """
+
+        prompt = f"""Given the following set of rules:
+
+                set_rules = {set_rules}
+
+                Some dict fields have special requirements listed below. First is the column header. After the - is the format. Do not include the instructions with your response:
+
+                requirements = {umich_all_asia_rules}
+
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+
+                input: {in_list[0]}
+
+                output: {out_list[0]}
+
+                input: {in_list[1]}
+
+                output: {out_list[1]}
+
+                input: {in_list[2]}
+
+                output: {out_list[2]}
+
+                input: {self.OCR}
+
+                output:"""
+        
+        return prompt
+    
+    def prompt_v1_palm2_noDomainKnowledge(self, OCR=None):
+        self.OCR = OCR or self.OCR
+        set_rules = """1. Your job is to return a new dict based on the structure of the reference dict ref_dict and these are your rules. 
+                        2. You must look at ref_dict and refactor the new text called OCR to match the same formatting. 
+                        3. OCR contains unstructured text inside of [], use your knowledge to put the OCR text into the correct ref_dict column. 
+                        4. If OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None" and skip all other steps.
+                        5. If there is a field that does not have a direct proxy in the OCR text, you can fill it in based on your knowledge, but you cannot generate new information.
+                        6. Never put text from the ref_dict values into the new dict, but you must use the headers from ref_dict. 
+                        7. There cannot be duplicate dictionary fields.
+                        8. Only return the new dict, do not explain your answer.
+                        9. Do not include quotation marks in content, only use quotation marks to represent values in dictionaries.
+                        10. For GPS coordinates only use Decimal Degrees (D.D°)
+                        11. "Given the input text, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values."""
+
+        umich_all_asia_rules = """
+        "Catalog Number" - {"format": "barcode", "null_value": "", "description": the barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits}
+        "Genus" - {"format": "Genus" or "Family indet" if no genus", "null_value": "", "description": taxonomic determination to genus, do captalize genus}
+        "Species"- {"format": "species" or "indet" if no species, "null_value": "", "description": taxonomic determination to species, do not captalize species}
+        "subspecies" - {"format": "subspecies", "null_value": "", "description": taxonomic determination to subspecies (subsp.)}
+        "variety" - {"format": "variety", "null_value": "", "description": taxonomic determination to variety (var)}
+        "forma" - {"format": "form", "null_value": "", "description": taxonomic determination to form (f.)}
+
+        "Country" - {"format": "Country", "null_value": "no data", "description": Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abreviation is given}
+        "State" - {"format": "Adm. Division 1", "null_value": "no data", "description": Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word}
+        "County" - {"format": "Adm. Division 2", "null_value": "no data", "description": Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word}
+        "Locality Name" - {"format": "verbatim", if no geographic info: "no data provided on label of catalog no: ######", or if illegible: "locality present but illegible/not translated for catalog no: #######", or if no named locality: "no named locality for catalog no: #######", "description": "Description of geographic location or landscape"}
+
+        "Min Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given}
+        "Max Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise}
+        "Elevation Units" - {format: "m", "null_value": "","description": "m" only if an elevation is present}
+        
+        "Verbatim Coordinates" - {"format": "Lat, Long, UTM, TRS", "null_value": "", "description": Convert coordinates to Decimal Degrees (D.D°) format, do not use Minutes, Seconds or quotation marks}
+
+        "Datum" - {"format": "WGS84, NAD23 etc.", "null_value": "not present", "description": Datum of coordinates on label; "" is GPS coordinates are not in OCR}
+        "Cultivated" - {"format": "yes", "null_value": "", "description": Indicates if specimen was grown in cultivation}
+        "Habitat" - {"format": "verbatim", "null_value": "", "description": Description of habitat or location where specimen was collected, ignore descriptions of the plant itself}
+        "Collectors" - {"format": "Collector", "null_value": "not present", "description": Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas}
+        "Collector Number" - {"format": "Collector No.", "null_value": "s.n.", "description": Sequential number assigned to collection, associated with the collector}
+        "Verbatim Date" - {"format": "verbatim", "null_value": "s.d.", "description": Date of collection exactly as it appears on the label}
+        "Date" - {"format": "yyyy-mm-dd", "null_value": "", "description": Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e. 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day}
+        "End Date" - {"format": "yyyy-mm-dd", "null_value": "", "description": If date range is listed, later date of collection range}
+        """
+        structure = """{
+            "Catalog Number": "",
+            "Genus": "",
+            "Species": "",
+            "subspecies": "",
+            "variety": "",
+            "forma": "",
+            "Country": "",
+            "State": "",
+            "County": "",
+            "Locality Name": "",
+            "Min Elevation": "",
+            "Max Elevation": "",
+            "Elevation Units": "",
+            "Verbatim Coordinates": "",
+            "Datum": "",
+            "Cultivated": "",
+            "Habitat": "",
+            "Collectors": "",
+            "Collector Number": "",
+            "Verbatim Date": "",
+            "Date": "",
+            "End Date": "",
+            }"""
+        # structure = """{
+        #             "Catalog Number": [Catalog Number],
+        #             "Genus": [Genus],
+        #             "Species": [species],
+        #             "subspecies": [subspecies],
+        #             "variety": [variety],
+        #             "forma": [forma],
+        #             "Country": [Country],
+        #             "State": [State],
+        #             "County": [County],
+        #             "Locality Name": [Locality Name],
+        #             "Min Elevation": [Min Elevation],
+        #             "Max Elevation": [Max Elevation],
+        #             "Elevation Units": [Elevation Units],
+        #             "Verbatim Coordinates": [Verbatim Coordinates],
+        #             "Datum": [Datum],
+        #             "Cultivated": [Cultivated],
+        #             "Habitat": [Habitat],
+        #             "Collectors": [Collectors],
+        #             "Collector Number": [Collector Number],
+        #             "Verbatim Date": [Verbatim Date],
+        #             "Date": [Date],
+        #             "End Date": [End Date]
+        #             }"""
+
+        prompt = f"""Given the following set of rules:
+                set_rules = {set_rules}
+                Some dict fields have special requirements listed below. First is the column header. After the - is the format. Do not include the instructions with your response:
+                requirements = {umich_all_asia_rules}
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+                The input unformatted OCR text: {self.OCR}
+                The output JSON structure: {structure}
+                The output JSON structure: {structure}
+                The output JSON structure: {structure}
+                The refactored JSON disctionary:"""
+        
+        return prompt
+    
+    def prompt_v2_palm2(self, OCR=None):
+        self.OCR = OCR or self.OCR
+        self.n_fields = 26 or self.n_fields
+
+        set_rules = """
+        1. Refactor the unstructured OCR text into a dictionary based on the JSON structure outlined below.
+        2. You should map the unstructured OCR text to the appropriate JSON key and then populate the field based on its rules.
+        3. Some JSON key fields are permitted to remain empty if the corresponding information is not found in the unstructured OCR text.
+        4. Ignore any information in the OCR text that doesn't fit into the defined JSON structure.
+        5. Duplicate dictionary fields are not allowed.
+        6. Ensure that all JSON keys are in lowercase.
+        7. Ensure that new JSON field values follow sentence case capitalization.
+        8. Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+        9. Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+        10. Only return a JSON dictionary represented as a string. You should not explain your answer.
+        """
+
+        dictionary_field_format_descriptions = """
+        The next section of instructions outlines how to format the JSON dictionary. The keys are the same as those of the final formatted JSON object.
+        For each key there is a format requirement that specifies how to transcribe the information for that key. 
+        The possible formatting options are:
+        1. "verbatim transcription" - field is populated with verbatim text from the unformatted OCR.
+        2. "spell check transcription" - field is populated with spelling corrected text from the unformatted OCR.
+        3. "boolean yes no" - field is populated with only yes or no.
+        4. "integer" - field is populated with only an integer.
+        5. "[list]" - field is populated from one of the values in the list.
+        6. "yyyy-mm-dd" - field is populated with a date in the format year-month-day.
+        The desired null value is also given. Populate the field with the null value of the information for that key is not present in the unformatted OCR text. 
+        """
+
+        json_template_rules = """
+            {
+                "catalog_number": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits."
+                },
+                "genus": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to genus. Genus must be capitalized. If genus is not present use the taxonomic family name followed by the word 'indet'."
+                },
+                "species": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to species, do not capitalize species."
+                },
+                "subspecies": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to subspecies (subsp.)."
+                },
+                "variety": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to variety (var)."
+                },
+                "forma": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Taxonomic determination to form (f.)."
+                },
+                "country": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Country that corresponds to the current geographic location of collection. Capitalize first letter of each word. If abbreviation is given populate field with the full spelling of the country's name. Use sentence-case to capitalize proper nouns."
+                },
+                "state": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Administrative division 1 that corresponds to the current geographic location of collection. Capitalize first letter of each word. Administrative division 1 is equivalent to a U.S. State. Use sentence-case to capitalize proper nouns."
+                },
+                "county": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word. Administrative division 2 is equivalent to a U.S. county, parish, borough. Use sentence-case to capitalize proper nouns."
+                },
+                "locality_name": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Description of geographic location, landscape, landmarks, regional features, nearby places, or any contextual information aiding in pinpointing the exact origin or site of the specimen. Use sentence-case to capitalize proper nouns."
+                },
+                "min_elevation": {
+                    "format": "integer",
+                    "null_value": "",
+                    "description": "Minimum elevation or altitude in meters. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
+                },
+                "max_elevation": {
+                    "format": "integer",
+                    "null_value": "",
+                    "description": "Maximum elevation or altitude in meters. If only one elevation is present, then max_elevation should be set to the null_value. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
+                },
+                "elevation_units": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Elevation units must be meters. If min_elevation field is populated, then elevation_units: 'm'. Otherwise elevation_units: ''"
+                },
+                "verbatim_coordinates": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Verbatim location coordinates as they appear on the label. Do not convert formats. Possible coordinate types are one of [Lat, Long, UTM, TRS]."
+                },
+                "decimal_coordinates": {
+                    "format": "spell check transcription",
+                    "null_value": "",
+                    "description": "Correct and convert the verbatim location coordinates to conform with the decimal degrees GPS coordinate format."
+                },
+                "datum": {
+                    "format": "[WGS84, WGS72, WGS66, WGS60, NAD83, NAD27, OSGB36, ETRS89, ED50, GDA94, JGD2011, Tokyo97, KGD2002, TWD67, TWD97, BJS54, XAS80, GCJ-02, BD-09, PZ-90.11, GTRF, CGCS2000, ITRF88, ITRF89, ITRF90, ITRF91, ITRF92, ITRF93, ITRF94, ITRF96, ITRF97, ITRF2000, ITRF2005, ITRF2008, ITRF2014, Hong Kong Principal Datum, SAD69]",
+                    "null_value": "",
+                    "description": "Datum of location coordinates. Possible values are include in the format list. Leave field blank if unclear."
+                },
+                "cultivated": {
+                    "format": "boolean yes no",
+                    "null_value": "",
+                    "description": "Cultivated plants are intentionally grown by humans. In text descriptions, look for planting dates, garden locations, ornamental, cultivar names, garden, or farm to indicate cultivated plant."
+                },
+                "habitat": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Description of a plant's habitat or the location where the specimen was collected. Ignore descriptions of the plant itself. Use sentence-case to capitalize proper nouns."
+                },
+                "plant_description": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Description of plant features such as leaf shape, size, color, stem texture, height, flower structure, scent, fruit or seed characteristics, root system type, overall growth habit and form, any notable aroma or secretions, presence of hairs or bristles, and any other distinguishing morphological or physiological characteristics. Use sentence-case to capitalize proper nouns."
+                },
+                "collectors": {
+                    "format": "verbatim transcription",
+                    "null_value": "not present",
+                    "description": "Full name(s) of the individual(s) responsible for collecting the specimen. Use sentence-case to capitalize proper nouns. When multiple collectors are involved, their names should be separated by commas."
+                },
+                "collector_number": {
+                    "format": "verbatim transcription",
+                    "null_value": "s.n.",
+                    "description": "Unique identifier or number that denotes the specific collecting event and associated with the collector."
+                },
+                "determined_by": {
+                    "format": "verbatim transcription",
+                    "null_value": "",
+                    "description": "Full name of the individual responsible for determining the taxanomic name of the specimen. Use sentence-case to capitalize proper nouns. Sometimes the name will be near to the characters 'det' to denote determination. This name may be isolated from other names in the unformatted OCR text."
+                },
+                "multiple_names": {
+                    "format": "boolean yes no",
+                    "null_value": "",
+                    "description": "Indicate whether multiple people or collector names are present in the unformatted OCR text. Use sentence-case to capitalize proper nouns. If you see more than one person's name the value is 'yes'; otherwise the value is 'no'."
+                },
+                "verbatim_date": {
+                    "format": "verbatim transcription",
+                    "null_value": "s.d.",
+                    "description": "Date of collection exactly as it appears on the label. Do not change the format or correct typos."
+                },
+                "date": {
+                    "format": "yyyy-mm-dd",
+                    "null_value": "",
+                    "description": "Date the specimen was collected formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire date is unknown, 'YYYY-00-00' if only the year is known, and 'YYYY-MM-00' if year and month are known but day is not."
+                },
+                "end_date": {
+                    "format": "yyyy-mm-dd",
+                    "null_value": "",
+                    "description": "If a date range is provided, this represents the later or ending date of the collection period, formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire end date is unknown, 'YYYY-00-00' if only the year of the end date is known, and 'YYYY-MM-00' if year and month of the end date are known but the day is not."
+                },
+            }"""
+   
+        structure = """{"catalog_number": "",
+                        "genus": "",
+                        "species": "".
+                        "subspecies": "",
+                        "variety": "",
+                        "forma":"",
+                        "country": "",
+                        "state": "",
+                        "county": "",
+                        "locality_name": "",
+                        "min_elevation": "",
+                        "max_elevation": "",
+                        "elevation_units": "',
+                        "verbatim_coordinates": "",
+                        "decimal_coordinates": "",
+                        "datum": "",
+                        "cultivated": "",
+                        "habitat": "",
+                        "plant_description": "",
+                        "collectors": "",
+                        "collector_number": "",
+                        "determined_by": "",
+                        "multiple_names": "',
+                        "verbatim_date": "",
+                        "date": "",
+                        "end_date": "",
+                        }"""
+        # structure = """{"catalog_number": [Catalog Number],
+        #                 "genus": [Genus],
+        #                 "species": [species],
+        #                 "subspecies": [subspecies],
+        #                 "variety": [variety],
+        #                 "forma": [forma],
+        #                 "country": [Country],
+        #                 "state": [State],
+        #                 "county": [County],
+        #                 "locality_name": [Locality Name],
+        #                 "min_elevation": [Min Elevation],
+        #                 "max_elevation": [Max Elevation],
+        #                 "elevation_units": [Elevation Units],
+        #                 "verbatim_coordinates": [Verbatim Coordinates],
+        #                 "decimal_coordinates": [Decimal Coordinates],
+        #                 "datum": [Datum],
+        #                 "cultivated": [boolean yes no],
+        #                 "habitat": [Habitat Description],
+        #                 "plant_description": [Plant Description],
+        #                 "collectors": [Name(s) of Collectors],
+        #                 "collector_number": [Collector Number],
+        #                 "determined_by": [Name(s) of Taxonomist],
+        #                 "multiple_names": [boolean yes no],
+        #                 "verbatim_date": [Verbatim Date],
+        #                 "date": [yyyy-mm-dd],
+        #                 "end_date": [yyyy-mm-dd],
+        #                 }"""
+
+        prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
+        The rules are:
+        {set_rules}
+        The unstructured OCR text is:
+        {self.OCR}
+        {dictionary_field_format_descriptions}
+        This is the JSON template that includes instructions for each key:
+        {json_template_rules}
+        Please populate the following JSON dictionary based on the rules and the unformatted OCR text. The square brackets denote the locations that you should place the new structured text:
+        {structure}
+        {structure}
+        {structure}
+        """
+        
+        return prompt
+    
+    def prompt_palm_redo_v1(self, incorrect_json):
+        structure = """{
+                    "Catalog Number": [Catalog Number],
+                    "Genus": [Genus],
+                    "Species": [species],
+                    "subspecies": [subspecies],
+                    "variety": [variety],
+                    "forma": [forma],
+                    "Country": [Country],
+                    "State": [State],
+                    "County": [County],
+                    "Locality Name": [Locality Name],
+                    "Min Elevation": [Min Elevation],
+                    "Max Elevation": [Max Elevation],
+                    "Elevation Units": [Elevation Units],
+                    "Verbatim Coordinates": [Verbatim Coordinates],
+                    "Datum": [Datum],
+                    "Cultivated": [Cultivated],
+                    "Habitat": [Habitat],
+                    "Collectors": [Collectors],
+                    "Collector Number": [Collector Number],
+                    "Verbatim Date": [Verbatim Date],
+                    "Date": [Date],
+                    "End Date": [End Date]
+                    }"""
+
+        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
+                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
+                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+                The incorrectly formatted JSON dictionary: {incorrect_json}
+                The output JSON structure: {structure}
+                The output JSON structure: {structure}
+                The output JSON structure: {structure}
+                The refactored JSON disctionary: """
+        return prompt
+        
+    def prompt_palm_redo_v2(self, incorrect_json):
+        structure = """{"catalog_number": "",
+                        "genus": "",
+                        "species": "".
+                        "subspecies": "",
+                        "variety": "",
+                        "forma":"",
+                        "country": "",
+                        "state": "",
+                        "county": "",
+                        "locality_name": "",
+                        "min_elevation": "",
+                        "max_elevation": "",
+                        "elevation_units": "',
+                        "verbatim_coordinates": "",
+                        "decimal_coordinates": "",
+                        "datum": "",
+                        "cultivated": "",
+                        "habitat": "",
+                        "plant_description": "",
+                        "collectors": "",
+                        "collector_number": "",
+                        "determined_by": "",
+                        "multiple_names": "',
+                        "verbatim_date": "",
+                        "date": "",
+                        "end_date": "",
+                        }"""
+        
+        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
+                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
+                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
+                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
+                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
+                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
+                The incorrectly formatted JSON dictionary: {incorrect_json}
+                The output JSON structure: {structure}
+                The output JSON structure: {structure}
+                The output JSON structure: {structure}
+                The refactored JSON disctionary: """
+        return prompt

vouchervision/LLM_GoogleGemini.py

@@ -0,0 +1,152 @@
+import os, time
+import vertexai
+from vertexai.preview.generative_models import GenerativeModel
+from vertexai.generative_models._generative_models import HarmCategory, HarmBlockThreshold
+from langchain.output_parsers import RetryWithErrorOutputParser
+from langchain.schema import HumanMessage
+from langchain.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser
+from langchain_google_genai import ChatGoogleGenerativeAI
+
+from vouchervision.utils_LLM import SystemLoadMonitor, count_tokens
+from vouchervision.utils_LLM_JSON_validation import validate_and_align_JSON_keys_with_template
+from vouchervision.utils_taxonomy_WFO import validate_taxonomy_WFO
+from vouchervision.utils_geolocate_HERE import validate_coordinates_here
+
+class GoogleGeminiHandler: 
+
+    RETRY_DELAY = 10  # Wait 10 seconds before retrying
+    MAX_RETRIES = 3  # Maximum number of retries
+    TOKENIZER_NAME = 'gpt-4'
+    VENDOR = 'google'
+    STARTING_TEMP = 0.5
+
+    def __init__(self, logger, model_name, JSON_dict_structure):
+        self.logger = logger
+        self.model_name = model_name
+        self.JSON_dict_structure = JSON_dict_structure
+
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp   
+
+        self.monitor = SystemLoadMonitor(logger)
+
+        self.parser = JsonOutputParser()
+
+        # Define the prompt template
+        self.prompt = PromptTemplate(
+            template="Answer the user query.\n{format_instructions}\n{query}\n",
+            input_variables=["query"],
+            partial_variables={"format_instructions": self.parser.get_format_instructions()},
+        )
+        self._set_config()
+
+
+    def _set_config(self):
+        # os.environ['GOOGLE_API_KEY'] # Must be set too for the retry call, set in VoucherVision class along with other API Keys
+        # vertexai.init(project=os.environ['PALM_PROJECT_ID'], location=os.environ['PALM_LOCATION'])
+        self.config = {
+                "max_output_tokens": 1024,
+                "temperature": self.starting_temp,
+                "top_p": 1
+            }
+        self.safety_settings = {
+            HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
+            HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
+            HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
+            HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
+        }
+        self._build_model_chain_parser()
+        
+    def _adjust_config(self):
+        new_temp = self.adjust_temp + self.temp_increment
+        self.json_report.set_text(text_main=f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.logger.info(f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.adjust_temp += self.temp_increment
+        self.config['temperature'] = self.adjust_temp   
+
+    def _reset_config(self):
+        self.json_report.set_text(text_main=f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.logger.info(f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.adjust_temp = self.starting_temp
+        self.config['temperature'] = self.starting_temp  
+
+    def _build_model_chain_parser(self):
+        # Instantiate the LLM class for Google Gemini
+        self.llm_model = ChatGoogleGenerativeAI(model='gemini-pro',
+                                    max_output_tokens=self.config.get('max_output_tokens'),
+                                    top_p=self.config.get('top_p'))    
+        # Set up the retry parser with the runnable
+        self.retry_parser = RetryWithErrorOutputParser.from_llm(parser=self.parser, llm=self.llm_model, max_retries=self.MAX_RETRIES)
+        # Prepare the chain
+        self.chain = self.prompt | self.call_google_gemini     
+
+    # Define a function to format the input for Google Gemini call
+    def call_google_gemini(self, prompt_text):
+        model = GenerativeModel(self.model_name)
+        response = model.generate_content(prompt_text.text,
+                                        generation_config=self.config,
+                                        safety_settings=self.safety_settings)
+        return response.text
+    
+    def call_llm_api_GoogleGemini(self, prompt_template, json_report):
+        self.json_report = json_report
+        self.json_report.set_text(text_main=f'Sending request to {self.model_name}')
+        self.monitor.start_monitoring_usage()
+        nt_in = 0
+        nt_out = 0
+        
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            try:
+                model_kwargs = {"temperature": self.adjust_temp}
+                # Invoke the chain to generate prompt text
+                response = self.chain.invoke({"query": prompt_template, "model_kwargs": model_kwargs})
+
+                # Use retry_parser to parse the response with retry logic
+                output = self.retry_parser.parse_with_prompt(response, prompt_value=prompt_template)
+
+                if output is None:
+                    self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response}')
+                    self._adjust_config()
+                else:
+                    nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                    nt_out = count_tokens(response, self.VENDOR, self.TOKENIZER_NAME)
+
+                    output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                    if output is None:
+                        self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response}')
+                        self._adjust_config() 
+                    else:
+                        json_report.set_text(text_main=f'Working on WFO and Geolocation')
+                        output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False) ###################################### make this configurable
+                        output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False) ###################################### make this configurable
+
+                        self.logger.info(f"Formatted JSON: {output}")
+                        
+                        self.monitor.stop_monitoring_report_usage()    
+
+                        if self.adjust_temp != self.starting_temp:            
+                            self._reset_config()
+
+                        json_report.set_text(text_main=f'LLM call successful')
+                        return output, nt_in, nt_out, WFO_record, GEO_record
+
+            except Exception as e:
+                self.logger.error(f'{e}')
+                
+                self._adjust_config()           
+                time.sleep(self.RETRY_DELAY)
+
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        self.json_report.set_text(text_main=f'Failed to extract valid JSON after [{ind}] attempts')
+
+        self.monitor.stop_monitoring_report_usage()                
+        self._reset_config()
+
+        json_report.set_text(text_main=f'LLM call failed')
+        return None, nt_in, nt_out, None, None
+
+

vouchervision/LLM_GooglePalm2.py

@@ -0,0 +1,162 @@
+import os, time
+import vertexai
+from vertexai.language_models import TextGenerationModel
+from vertexai.generative_models._generative_models import HarmCategory, HarmBlockThreshold
+from langchain.output_parsers import RetryWithErrorOutputParser
+from langchain.schema import HumanMessage
+from langchain.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser
+from langchain_google_genai import ChatGoogleGenerativeAI
+
+from vouchervision.utils_LLM import SystemLoadMonitor, count_tokens
+from vouchervision.utils_LLM_JSON_validation import validate_and_align_JSON_keys_with_template
+from vouchervision.utils_taxonomy_WFO import validate_taxonomy_WFO
+from vouchervision.utils_geolocate_HERE import validate_coordinates_here
+
+#https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk
+#pip install --upgrade google-cloud-aiplatform
+# from google.cloud import aiplatform
+
+#### have to authenticate gcloud 
+# gcloud auth login
+# gcloud config set project XXXXXXXXX
+# https://cloud.google.com/docs/authentication
+
+class GooglePalm2Handler: 
+
+    RETRY_DELAY = 10  # Wait 10 seconds before retrying
+    MAX_RETRIES = 3  # Maximum number of retries
+    TOKENIZER_NAME = 'gpt-4'
+    VENDOR = 'google'
+    STARTING_TEMP = 0.5
+
+    def __init__(self, logger, model_name, JSON_dict_structure):
+        self.logger = logger
+        self.model_name = model_name
+        self.JSON_dict_structure = JSON_dict_structure
+
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp   
+
+        self.monitor = SystemLoadMonitor(logger)
+
+        self.parser = JsonOutputParser()
+
+        # Define the prompt template
+        self.prompt = PromptTemplate(
+            template="Answer the user query.\n{format_instructions}\n{query}\n",
+            input_variables=["query"],
+            partial_variables={"format_instructions": self.parser.get_format_instructions()},
+        )
+        self._set_config()
+
+    def _set_config(self):
+        # vertexai.init(project=os.environ['PALM_PROJECT_ID'], location=os.environ['PALM_LOCATION'])
+        self.config = {
+                "max_output_tokens": 1024,
+                "temperature": self.starting_temp,
+                "top_p": 1
+            }
+        self.safety_settings = {
+            HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
+            HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
+            HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
+            HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
+        }
+        self._build_model_chain_parser()
+
+    def _adjust_config(self):
+        new_temp = self.adjust_temp + self.temp_increment
+        self.json_report.set_text(text_main=f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.logger.info(f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.adjust_temp += self.temp_increment
+        self.config['temperature'] = self.adjust_temp    
+
+    def _reset_config(self):
+        self.json_report.set_text(text_main=f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.logger.info(f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.adjust_temp = self.starting_temp
+        self.config['temperature'] = self.starting_temp   
+
+    def _build_model_chain_parser(self):
+        # Instantiate the parser and the retry parser
+        self.llm_model = ChatGoogleGenerativeAI(model=self.model_name, 
+                                                max_output_tokens=self.config.get('max_output_tokens'),
+                                                top_p=self.config.get('top_p'))
+        
+        self.retry_parser = RetryWithErrorOutputParser.from_llm(
+                                                parser=self.parser,
+                                                llm=self.llm_model,
+                                                max_retries=self.MAX_RETRIES)
+        # Prepare the chain
+        self.chain = self.prompt | self.call_google_palm2
+
+    # Define a function to format the input for Google PaLM call
+    def call_google_palm2(self, prompt_text):
+        model = TextGenerationModel.from_pretrained(self.model_name)
+        response = model.predict(prompt_text.text,
+                                max_output_tokens=self.config.get('max_output_tokens'),
+                                temperature=self.config.get('temperature'),
+                                top_p=self.config.get('top_p'))
+        return response.text
+
+
+    def call_llm_api_GooglePalm2(self, prompt_template, json_report):
+        self.json_report = json_report
+        self.json_report.set_text(text_main=f'Sending request to {self.model_name}')
+        self.monitor.start_monitoring_usage()
+        nt_in = 0
+        nt_out = 0
+
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            try:
+                model_kwargs = {"temperature": self.adjust_temp}
+                # Invoke the chain to generate prompt text
+                response = self.chain.invoke({"query": prompt_template, "model_kwargs": model_kwargs})
+
+                # Use retry_parser to parse the response with retry logic
+                output = self.retry_parser.parse_with_prompt(response, prompt_value=prompt_template)
+
+                if output is None:
+                    self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response}')
+                    self._adjust_config()
+                else:
+                    nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                    nt_out = count_tokens(response, self.VENDOR, self.TOKENIZER_NAME)
+                    
+                    output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                    if output is None:
+                        self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response}')
+                        self._adjust_config()           
+                    else:
+                        json_report.set_text(text_main=f'Working on WFO and Geolocation')
+                        output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False) ###################################### make this configurable
+                        output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False) ###################################### make this configurable
+
+                        self.logger.info(f"Formatted JSON: {output}")
+                        
+                        self.monitor.stop_monitoring_report_usage()    
+
+                        if self.adjust_temp != self.starting_temp:            
+                            self._reset_config()
+
+                        json_report.set_text(text_main=f'LLM call successful')
+                        return output, nt_in, nt_out, WFO_record, GEO_record
+
+            except Exception as e:
+                self.logger.error(f'{e}')
+                
+                self._adjust_config()           
+                time.sleep(self.RETRY_DELAY)
+
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        self.json_report.set_text(text_main=f'Failed to extract valid JSON after [{ind}] attempts')
+
+        self.monitor.stop_monitoring_report_usage()                
+        self._reset_config()
+
+        json_report.set_text(text_main=f'LLM call failed')
+        return None, nt_in, nt_out, None, None

vouchervision/LLM_MistralAI.py

@@ -0,0 +1,139 @@
+import os, time, random, torch
+from langchain_mistralai.chat_models import ChatMistralAI
+from langchain.output_parsers import RetryWithErrorOutputParser
+from langchain.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser
+
+from vouchervision.utils_LLM import SystemLoadMonitor, count_tokens
+from vouchervision.utils_LLM_JSON_validation import validate_and_align_JSON_keys_with_template
+from vouchervision.utils_taxonomy_WFO import validate_taxonomy_WFO
+from vouchervision.utils_geolocate_HERE import validate_coordinates_here
+
+
+class MistralHandler: 
+    RETRY_DELAY = 2  # Wait 10 seconds before retrying
+    MAX_RETRIES = 5  # Maximum number of retries
+    STARTING_TEMP = 0.1
+    TOKENIZER_NAME = None
+    VENDOR = 'mistral'
+    RANDOM_SEED = 2023
+
+    def __init__(self, logger, model_name, JSON_dict_structure):
+        self.logger = logger
+        self.monitor = SystemLoadMonitor(logger)
+        self.has_GPU = torch.cuda.is_available()        
+        self.model_name = model_name
+        self.JSON_dict_structure = JSON_dict_structure
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp 
+
+        # Set up a parser
+        self.parser = JsonOutputParser()
+
+        # Define the prompt template
+        self.prompt = PromptTemplate(
+            template="Answer the user query.\n{format_instructions}\n{query}\n",
+            input_variables=["query"],
+            partial_variables={"format_instructions": self.parser.get_format_instructions()},
+        )
+
+        self._set_config()
+
+    def _set_config(self):
+        self.config = {'max_tokens': 1024,
+                'temperature': self.starting_temp,
+                'random_seed': self.RANDOM_SEED,
+                'safe_mode': False,
+                'top_p': 1,
+                }
+        self._build_model_chain_parser()
+
+
+    def _adjust_config(self):
+        new_temp = self.adjust_temp + self.temp_increment
+        self.config['random_seed'] = random.randint(1, 1000) 
+        self.json_report.set_text(text_main=f'Incrementing temperature from {self.adjust_temp} to {new_temp} and random_seed to {self.config.get("random_seed")}')
+        self.logger.info(f'Incrementing temperature from {self.adjust_temp} to {new_temp} and random_seed to {self.config.get("random_seed")}')
+        self.adjust_temp += self.temp_increment
+        self.config['temperature'] = self.adjust_temp    
+
+    def _reset_config(self):
+        self.json_report.set_text(text_main=f'Resetting temperature from {self.adjust_temp} to {self.starting_temp} and random_seed to {self.RANDOM_SEED}')
+        self.logger.info(f'Incrementing temperature from {self.adjust_temp} to {self.starting_temp} and random_seed to {self.RANDOM_SEED}')
+        self.adjust_temp = self.starting_temp
+        self.config['temperature'] = self.starting_temp    
+        self.config['random_seed'] = self.RANDOM_SEED
+          
+    def _build_model_chain_parser(self):
+        # Initialize MistralAI
+        self.llm_model = ChatMistralAI(mistral_api_key=os.environ.get("MISTRAL_API_KEY"),
+                            model=self.model_name,
+                            max_tokens=self.config.get('max_tokens'), 
+                            safe_mode=self.config.get('safe_mode'), 
+                            top_p=self.config.get('top_p'))
+        
+        # Set up the retry parser with the runnable
+        self.retry_parser = RetryWithErrorOutputParser.from_llm(parser=self.parser, llm=self.llm_model, max_retries=self.MAX_RETRIES)
+        
+        self.chain = self.prompt | self.llm_model    
+
+    def call_llm_api_MistralAI(self, prompt_template, json_report):
+        self.json_report = json_report
+        self.json_report.set_text(text_main=f'Sending request to {self.model_name}')
+        self.monitor.start_monitoring_usage()
+        nt_in = 0
+        nt_out = 0
+
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            try:
+                model_kwargs = {"temperature": self.adjust_temp, "random_seed": self.config.get("random_seed")}
+                
+                # Invoke the chain to generate prompt text
+                response = self.chain.invoke({"query": prompt_template, "model_kwargs": model_kwargs})
+
+                # Use retry_parser to parse the response with retry logic
+                output = self.retry_parser.parse_with_prompt(response.content, prompt_value=prompt_template)
+
+                if output is None:
+                    self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response}')
+                    self._adjust_config()
+                else:
+                    nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                    nt_out = count_tokens(response.content, self.VENDOR, self.TOKENIZER_NAME)
+                    
+                    output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                    if output is None:
+                        self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response}')
+                        self._adjust_config()           
+                    else:
+                        json_report.set_text(text_main=f'Working on WFO and Geolocation')
+                        output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False) ###################################### make this configurable
+                        output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False) ###################################### make this configurable
+
+                        self.logger.info(f"Formatted JSON: {output}")
+                        
+                        self.monitor.stop_monitoring_report_usage()    
+
+                        if self.adjust_temp != self.starting_temp:            
+                            self._reset_config()
+
+                        json_report.set_text(text_main=f'LLM call successful')
+                        return output, nt_in, nt_out, WFO_record, GEO_record
+
+            except Exception as e:
+                self.logger.error(f'{e}')
+                
+                self._adjust_config()           
+                time.sleep(self.RETRY_DELAY)
+
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        self.json_report.set_text(text_main=f'Failed to extract valid JSON after [{ind}] attempts')
+
+        self.monitor.stop_monitoring_report_usage()                
+        self._reset_config()
+        json_report.set_text(text_main=f'LLM call failed')
+
+        return None, nt_in, nt_out, None, None

vouchervision/LLM_OpenAI.py

@@ -0,0 +1,160 @@
+import time, torch
+from langchain.prompts import PromptTemplate
+from langchain_openai import ChatOpenAI, OpenAI
+from langchain.schema import HumanMessage
+from langchain_core.output_parsers import JsonOutputParser
+from langchain.output_parsers import RetryWithErrorOutputParser
+
+from vouchervision.utils_LLM import SystemLoadMonitor, count_tokens
+from vouchervision.utils_LLM_JSON_validation import validate_and_align_JSON_keys_with_template
+from vouchervision.utils_taxonomy_WFO import validate_taxonomy_WFO
+from vouchervision.utils_geolocate_HERE import validate_coordinates_here
+
+class OpenAIHandler: 
+    RETRY_DELAY = 10  # Wait 10 seconds before retrying
+    MAX_RETRIES = 3  # Maximum number of retries
+    STARTING_TEMP = 0.5
+    TOKENIZER_NAME = 'gpt-4'
+    VENDOR = 'openai'
+
+    def __init__(self, logger, model_name, JSON_dict_structure, is_azure, llm_object):
+        self.logger = logger
+        self.model_name = model_name
+        self.JSON_dict_structure = JSON_dict_structure
+        self.is_azure = is_azure
+        self.llm_object = llm_object
+        self.name_parts = self.model_name.split('-')
+        
+        self.monitor = SystemLoadMonitor(logger)
+        self.has_GPU = torch.cuda.is_available() 
+
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp 
+        
+        # Set up a parser
+        self.parser = JsonOutputParser()
+
+        self.prompt = PromptTemplate(
+            template="Answer the user query.\n{format_instructions}\n{query}\n",
+            input_variables=["query"],
+            partial_variables={"format_instructions": self.parser.get_format_instructions()},
+        )
+        self._set_config()
+
+    def _set_config(self):
+        self.config = {'max_new_tokens': 1024,
+                'temperature': self.starting_temp,
+                'random_seed': 2023,
+                'top_p': 1,
+                }
+        # Adjusting the LLM settings based on whether Azure is used
+        if self.is_azure:
+            self.llm_object.deployment_name = self.model_name
+            self.llm_object.model_name = self.model_name
+        else:
+            self.llm_object = None
+        self._build_model_chain_parser()
+
+
+       # Define a function to format the input for azure_call
+    def format_input_for_azure(self, prompt_text):
+        msg = HumanMessage(content=prompt_text.text)
+        # self.llm_object.temperature = self.config.get('temperature')
+        return self.llm_object(messages=[msg]) 
+
+    def _adjust_config(self):
+        new_temp = self.adjust_temp + self.temp_increment
+        self.json_report.set_text(text_main=f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.logger.info(f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.adjust_temp += self.temp_increment
+        self.config['temperature'] = self.adjust_temp   
+
+    def _reset_config(self):
+        self.json_report.set_text(text_main=f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.logger.info(f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.adjust_temp = self.starting_temp
+        self.config['temperature'] = self.starting_temp   
+        
+    def _build_model_chain_parser(self):
+        if not self.is_azure and ('instruct' in self.name_parts):
+            # Set up the retry parser with 3 retries
+            self.retry_parser = RetryWithErrorOutputParser.from_llm(
+                # parser=self.parser, llm=self.llm_object if self.is_azure else OpenAI(temperature=self.config.get('temperature'), model=self.model_name), max_retries=self.MAX_RETRIES
+                parser=self.parser, llm=self.llm_object if self.is_azure else OpenAI(model=self.model_name), max_retries=self.MAX_RETRIES
+            )
+        else:
+            # Set up the retry parser with 3 retries
+            self.retry_parser = RetryWithErrorOutputParser.from_llm(
+                # parser=self.parser, llm=self.llm_object if self.is_azure else ChatOpenAI(temperature=self.config.get('temperature'), model=self.model_name), max_retries=self.MAX_RETRIES
+                parser=self.parser, llm=self.llm_object if self.is_azure else ChatOpenAI(model=self.model_name), max_retries=self.MAX_RETRIES
+            )
+        # Prepare the chain
+        if not self.is_azure and ('instruct' in self.name_parts):
+            # self.chain = self.prompt | (self.format_input_for_azure if self.is_azure else OpenAI(temperature=self.config.get('temperature'), model=self.model_name))
+            self.chain = self.prompt | (self.format_input_for_azure if self.is_azure else OpenAI(model=self.model_name))
+        else:
+            # self.chain = self.prompt | (self.format_input_for_azure if self.is_azure else ChatOpenAI(temperature=self.config.get('temperature'), model=self.model_name))
+            self.chain = self.prompt | (self.format_input_for_azure if self.is_azure else ChatOpenAI(model=self.model_name))
+
+
+    def call_llm_api_OpenAI(self, prompt_template, json_report):
+        self.json_report = json_report
+        self.json_report.set_text(text_main=f'Sending request to {self.model_name}')
+        self.monitor.start_monitoring_usage()
+        nt_in = 0
+        nt_out = 0
+        
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            try:
+                model_kwargs = {"temperature": self.adjust_temp}
+                # Invoke the chain to generate prompt text
+                response = self.chain.invoke({"query": prompt_template, "model_kwargs": model_kwargs})
+
+                response_text = response.content if not isinstance(response, str) else response
+
+                # Use retry_parser to parse the response with retry logic
+                output = self.retry_parser.parse_with_prompt(response_text, prompt_value=prompt_template)
+
+                if output is None:
+                    self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response_text}')
+                    self._adjust_config()
+                else:
+                    nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                    nt_out = count_tokens(response_text, self.VENDOR, self.TOKENIZER_NAME)
+                
+                    output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                    if output is None:
+                        self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{response_text}')
+                        self._adjust_config()   
+                    else:
+                        json_report.set_text(text_main=f'Working on WFO and Geolocation')
+
+                        output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False) ###################################### make this configurable
+                        output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False) ###################################### make this configurable
+
+                        self.logger.info(f"Formatted JSON: {output}")
+
+                        self.monitor.stop_monitoring_report_usage()    
+                        
+                        if self.adjust_temp != self.starting_temp:            
+                            self._reset_config()
+                        json_report.set_text(text_main=f'LLM call successful')
+                        return output, nt_in, nt_out, WFO_record, GEO_record
+            
+            except Exception as e:
+                self.logger.error(f'{e}')
+                
+                self._adjust_config()           
+                time.sleep(self.RETRY_DELAY)
+
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        self.json_report.set_text(text_main=f'Failed to extract valid JSON after [{ind}] attempts')
+
+        self.monitor.stop_monitoring_report_usage()                
+        self._reset_config()
+
+        json_report.set_text(text_main=f'LLM call failed')
+        return None, nt_in, nt_out, None, None

vouchervision/LLM_PaLM.py

@@ -1,209 +0,0 @@
-import os
-import sys
-import inspect
-import json
-from json import JSONDecodeError
-import tiktoken
-import random 
-import google.generativeai as palm
-
-currentdir = os.path.dirname(os.path.abspath(
-    inspect.getfile(inspect.currentframe())))
-parentdir = os.path.dirname(currentdir)
-sys.path.append(parentdir)
-
-from prompt_catalog import PromptCatalog
-from general_utils import num_tokens_from_string
-
-"""
-DEPRECATED:
-    Safety setting regularly block a response, so set to 4 to disable
-
-    class HarmBlockThreshold(Enum):
-        HARM_BLOCK_THRESHOLD_UNSPECIFIED = 0
-        BLOCK_LOW_AND_ABOVE = 1
-        BLOCK_MEDIUM_AND_ABOVE = 2
-        BLOCK_ONLY_HIGH = 3
-        BLOCK_NONE = 4
-"""
-
-SAFETY_SETTINGS = [
-    {
-        "category": "HARM_CATEGORY_DEROGATORY",
-        "threshold": "BLOCK_NONE",
-    },
-    {
-        "category": "HARM_CATEGORY_TOXICITY",
-        "threshold": "BLOCK_NONE",
-    },
-    {
-        "category": "HARM_CATEGORY_VIOLENCE",
-        "threshold": "BLOCK_NONE",
-    },
-    {
-        "category": "HARM_CATEGORY_SEXUAL",
-        "threshold": "BLOCK_NONE",
-    },
-    {
-        "category": "HARM_CATEGORY_MEDICAL",
-        "threshold": "BLOCK_NONE",
-    },
-    {
-        "category": "HARM_CATEGORY_DANGEROUS",
-        "threshold": "BLOCK_NONE",
-    },
-]
-
-PALM_SETTINGS = {
-    'model': 'models/text-bison-001',
-    'temperature': 0,
-    'candidate_count': 1,
-    'top_k': 40,
-    'top_p': 0.95,
-    'max_output_tokens': 8000,
-    'stop_sequences': [],
-    'safety_settings': SAFETY_SETTINGS,
-}
-
-PALM_SETTINGS_REDO = {
-    'model': 'models/text-bison-001',
-    'temperature': 0.05,
-    'candidate_count': 1,
-    'top_k': 40,
-    'top_p': 0.95,
-    'max_output_tokens': 8000,
-    'stop_sequences': [],
-    'safety_settings': SAFETY_SETTINGS,
-}
-
-def OCR_to_dict_PaLM(logger, OCR, prompt_version, VVE):
-    try:
-        logger.info(f'Length of OCR raw -- {len(OCR)}')
-    except:
-        print(f'Length of OCR raw -- {len(OCR)}')
-        
-    # prompt = PROMPT_PaLM_UMICH_skeleton_all_asia(OCR, in_list, out_list) # must provide examples to PaLM differently than for chatGPT, at least 2 examples
-    Prompt = PromptCatalog(OCR) 
-    if prompt_version in ['prompt_v2_palm2']:
-        version = 'v2'
-        prompt = Prompt.prompt_v2_palm2(OCR)
-    
-    elif prompt_version in ['prompt_v1_palm2',]:
-        version = 'v1'
-        # create input: output: for PaLM
-        # Find a similar example from the domain knowledge
-        domain_knowledge_example = VVE.query_db(OCR, 4)
-        similarity= VVE.get_similarity()
-        domain_knowledge_example_string = json.dumps(domain_knowledge_example)
-        in_list, out_list = create_OCR_analog_for_input(domain_knowledge_example)
-        prompt = Prompt.prompt_v1_palm2(in_list, out_list, OCR)
-
-    elif prompt_version in ['prompt_v1_palm2_noDomainKnowledge',]:
-        version = 'v1'
-        prompt = Prompt.prompt_v1_palm2_noDomainKnowledge(OCR)
-    else:
-        version = 'custom'
-        prompt, n_fields, xlsx_headers = Prompt.prompt_v2_custom(prompt_version, OCR=OCR, is_palm=True)
-        # raise
-
-    nt = num_tokens_from_string(prompt, "cl100k_base")
-    # try:
-    logger.info(f'Prompt token length --- {nt}')
-    # except:
-        # print(f'Prompt token length --- {nt}')
-
-    do_use_SOP = False ########
-
-    if do_use_SOP:
-        '''TODO: Check back later to see if LangChain will support PaLM'''
-        # logger.info(f'Waiting for PaLM API call --- Using StructuredOutputParser')
-        # response = structured_output_parser(OCR, prompt, logger)
-        # return response['Dictionary']
-        pass
-
-    else:
-        # try:
-        logger.info(f'Waiting for PaLM 2 API call')
-        # except:
-            # print(f'Waiting for PaLM 2 API call --- Content')
-
-        # safety_thresh = 4
-        # PaLM_settings = {'model': 'models/text-bison-001','temperature': 0,'candidate_count': 1,'top_k': 40,'top_p': 0.95,'max_output_tokens': 8000,'stop_sequences': [],
-                        #  'safety_settings': [{"category":"HARM_CATEGORY_DEROGATORY","threshold":safety_thresh},{"category":"HARM_CATEGORY_TOXICITY","threshold":safety_thresh},{"category":"HARM_CATEGORY_VIOLENCE","threshold":safety_thresh},{"category":"HARM_CATEGORY_SEXUAL","threshold":safety_thresh},{"category":"HARM_CATEGORY_MEDICAL","threshold":safety_thresh},{"category":"HARM_CATEGORY_DANGEROUS","threshold":safety_thresh}],}
-        response = palm.generate_text(prompt=prompt, **PALM_SETTINGS)
-
-
-        if response and response.result:
-            if isinstance(response.result, (str, bytes)):
-                response_valid = check_and_redo_JSON(response, logger, version)
-            else:
-                response_valid = {}
-        else:
-            response_valid = {}
-
-        logger.info(f'Candidate JSON\n{response.result}')
-        return response_valid, nt
-
-def check_and_redo_JSON(response, logger, version):
-    try:
-        response_valid = json.loads(response.result)
-        logger.info(f'Response --- First call passed')
-        return response_valid
-    except JSONDecodeError:
-
-        try:
-            response_valid = json.loads(response.result.strip('```').replace('json\n', '', 1).replace('json', '', 1))
-            logger.info(f'Response --- Manual removal of ```json succeeded')
-            return response_valid
-        except:
-            logger.info(f'Response --- First call failed. Redo...')
-            Prompt = PromptCatalog() 
-            if version == 'v1':
-                prompt_redo = Prompt.prompt_palm_redo_v1(response.result)
-            elif version == 'v2':
-                prompt_redo = Prompt.prompt_palm_redo_v2(response.result)
-            elif version == 'custom':
-                prompt_redo = Prompt.prompt_v2_custom_redo(response.result, is_palm=True)
-
-
-            # prompt_redo = PROMPT_PaLM_Redo(response.result)
-            try:
-                response = palm.generate_text(prompt=prompt_redo, **PALM_SETTINGS)
-                response_valid = json.loads(response.result)
-                logger.info(f'Response --- Second call passed')
-                return response_valid
-            except JSONDecodeError:
-                logger.info(f'Response --- Second call failed. Final redo. Temperature changed to 0.05')
-                try:
-                    response = palm.generate_text(prompt=prompt_redo, **PALM_SETTINGS_REDO)
-                    response_valid = json.loads(response.result)
-                    logger.info(f'Response --- Third call passed')
-                    return response_valid
-                except JSONDecodeError:
-                    return None
-            
-
-def create_OCR_analog_for_input(domain_knowledge_example):
-    in_list = []
-    out_list = []
-    # Iterate over the domain_knowledge_example (list of dictionaries)
-    for row_dict in domain_knowledge_example:
-        # Convert the dictionary to a JSON string and add it to the out_list
-        domain_knowledge_example_string = json.dumps(row_dict)
-        out_list.append(domain_knowledge_example_string)
-
-        # Create a single string from all values in the row_dict
-        row_text = '||'.join(str(v) for v in row_dict.values())
-
-        # Split the row text by '||', shuffle the parts, and then re-join with a single space
-        parts = row_text.split('||')
-        random.shuffle(parts)
-        shuffled_text = ' '.join(parts)
-
-        # Add the shuffled_text to the in_list
-        in_list.append(shuffled_text)
-    return in_list, out_list
-
-
-def strip_problematic_chars(s):
-    return ''.join(c for c in s if c.isprintable())

vouchervision/LLM_chatGPT_3_5.py

@@ -1,427 +0,0 @@
-import openai
-import os, json, sys, inspect, time, requests
-from langchain.output_parsers import StructuredOutputParser, ResponseSchema
-from langchain.prompts import PromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate
-from langchain.llms import OpenAI
-from langchain.chat_models import ChatOpenAI, AzureChatOpenAI
-from langchain.schema import HumanMessage
-from general_utils import num_tokens_from_string
-
-currentdir = os.path.dirname(os.path.abspath(
-    inspect.getfile(inspect.currentframe())))
-parentdir = os.path.dirname(currentdir)
-sys.path.append(parentdir)
-
-from prompts import PROMPT_UMICH_skeleton_all_asia, PROMPT_OCR_Organized, PROMPT_UMICH_skeleton_all_asia_GPT4, PROMPT_OCR_Organized_GPT4, PROMPT_JSON
-from prompt_catalog import PromptCatalog
-
-RETRY_DELAY = 61  # Wait 60 seconds before retrying
-MAX_RETRIES = 5  # Maximum number of retries
-
-
-def azure_call(model, messages):
-    response = model(messages=messages)
-    return response
-
-def OCR_to_dict(is_azure, logger, MODEL, prompt, llm, prompt_version):
-    for i in range(MAX_RETRIES):
-        try:
-            do_use_SOP = True
-
-            if do_use_SOP:
-                logger.info(f'Waiting for {MODEL} API call --- Using StructuredOutputParser')
-                response = structured_output_parser(is_azure, MODEL, llm, prompt, logger, prompt_version)
-                if response is None:
-                    return None
-                else:
-                    return response['Dictionary']
-
-            else:
-                ### Direct GPT ###
-                logger.info(f'Waiting for {MODEL} API call')
-                if not is_azure:
-                    response = openai.ChatCompletion.create(
-                        model=MODEL,
-                        temperature = 0,
-                        messages=[
-                            {"role": "system", "content": "You are a helpful assistant acting as a transcription expert and your job is to transcribe herbarium specimen labels based on OCR data and reformat it to meet Darwin Core Archive Standards into a Python dictionary based on certain rules."},
-                            {"role": "user", "content": prompt},
-                        ],
-                        response_format={type: "json_object"},#################################### TODO ###############################################################################
-                        max_tokens=4096,
-                    )
-                    # print the model's response
-                    return response.choices[0].message['content']
-                else:
-                    msg = HumanMessage(
-                        content=prompt
-                    )
-                    response = azure_call(llm, [msg])
-                    return response.content
-        except Exception as e:
-            logger.error(f'{e}')
-            if i < MAX_RETRIES - 1:  # No delay needed after the last try
-                time.sleep(RETRY_DELAY)
-            else:
-                raise
-
-# def OCR_to_dict(logger, MODEL, prompt, OCR, BASE_URL, HEADERS):
-#     for i in range(MAX_RETRIES):
-#         try:
-#             do_use_SOP = False
-
-#             if do_use_SOP:
-#                 logger.info(f'Waiting for {MODEL} API call --- Using StructuredOutputParser -- Content')
-#                 response = structured_output_parser(MODEL, OCR, prompt, logger)
-#                 if response is None:
-#                     return None
-#                 else:
-#                     return response['Dictionary']
-
-#             else:
-#                 ### Direct GPT through Azure ###
-#                 logger.info(f'Waiting for {MODEL} API call')
-#                 response = azure_gpt_request(prompt, BASE_URL, HEADERS, model_name=MODEL)
-
-#                 # Handle the response data. Note: You might need to adjust the following line based on the exact response format of the Azure API.
-#                 content = response.get("choices", [{}])[0].get("message", {}).get("content", "")
-#                 return content
-#         except requests.exceptions.RequestException as e:  # Replace openai.error.APIError with requests exception.
-#             # Handle HTTP exceptions. You can adjust this based on the Azure API's error responses.
-#             if e.response.status_code == 502:
-#                 logger.info(f'   ***    502 error was encountered, wait and try again   ***')
-#                 if i < MAX_RETRIES - 1:
-#                     time.sleep(RETRY_DELAY)
-#             else:
-#                 raise
-
-
-def OCR_to_dict_16k(is_azure, logger, MODEL, prompt, llm, prompt_version):
-    for i in range(MAX_RETRIES):
-        try:
-            fs = FunctionSchema()
-            response = openai.ChatCompletion.create(
-                model=MODEL,
-                temperature = 0,
-                messages=[
-                    {"role": "system", "content": "You are a helpful assistant acting as a transcription expert and your job is to transcribe herbarium specimen labels based on OCR data and reformat it to meet Darwin Core Archive Standards into a Python dictionary based on certain rules."},
-                    {"role": "user", "content": prompt},
-                ],
-                max_tokens=8000,
-                function_call= "none",
-                functions= fs.format_C21_AA_V1()
-
-            )
-            # Try to parse the response into JSON
-            call_failed = False
-            try:
-                response_string = response.choices[0].message['content']
-            except:
-                call_failed = True
-                response_string = prompt
-
-            if not call_failed:
-                try:
-                    # Try to parse the response into JSON
-                    response_dict = json.loads(response_string)
-                    return response_dict['Dictionary']
-                except json.JSONDecodeError:
-                    # If the response is not a valid JSON, call the structured_output_parser_for_function_calls_fail function
-                    logger.info(f'Invalid JSON response, calling structured_output_parser_for_function_calls_fail function')
-                    logger.info(f'Waiting for {MODEL} API call --- Using StructuredOutputParser --- JSON Fixer')
-                    response_sop = structured_output_parser_for_function_calls_fail(is_azure, MODEL, response_string, logger, llm, prompt_version, is_helper=False)
-                    if response_sop is None:
-                        return None
-                    else:
-                        return response_sop['Dictionary']
-            else:
-                try:
-                    logger.info(f'Call Failed. Attempting fallback JSON parse without guidance')
-                    logger.info(f'Waiting for {MODEL} API call --- Using StructuredOutputParser --- JSON Fixer')
-                    response_sop = structured_output_parser_for_function_calls_fail(is_azure, MODEL, response_string, logger, llm, prompt_version, is_helper=False)
-                    if response_sop is None:
-                        return None
-                    else:
-                        return response_sop['Dictionary']
-                except:
-                    return None
-        except Exception as e:
-            # if e.status_code == 401: # or you can check the error message
-            logger.info(f'   ***    401 error was encountered, wait and try again   ***')
-            # If a 401 error was encountered, wait and try again
-            if i < MAX_RETRIES - 1:  # No delay needed after the last try
-                time.sleep(RETRY_DELAY)
-            else:
-                # If it was a different error, re-raise it
-                raise
-            
-def structured_output_parser(is_azure, MODEL, llm, prompt_template, logger, prompt_version, is_helper=False):
-    if not is_helper:
-        response_schemas = [
-            ResponseSchema(name="SpeciesName", description="Taxonomic determination, genus_species"),
-            ResponseSchema(name="Dictionary", description='Formatted JSON object'),]#prompt_template),]
-    elif is_helper:
-        response_schemas = [
-            ResponseSchema(name="Dictionary", description='Formatted JSON object'),#prompt_template),
-            ResponseSchema(name="Summary", description="A one sentence summary of the content"),]
-        
-    output_parser = StructuredOutputParser.from_response_schemas(response_schemas)
-
-    format_instructions = output_parser.get_format_instructions()
-
-    prompt = ChatPromptTemplate(
-        messages=[
-            HumanMessagePromptTemplate.from_template("Parse the OCR text into the correct structured format.\n{format_instructions}\n{question}")  
-        ],
-        input_variables=["question"],
-        partial_variables={"format_instructions": format_instructions}
-    )
-
-    # Handle Azure vs OpenAI implementation
-    if is_azure:
-        _input = prompt.format_prompt(question=prompt_template)
-        msg = HumanMessage(content=_input.to_string())
-        output = azure_call(llm, [msg])
-    else:
-        chat_model = ChatOpenAI(temperature=0, model=MODEL)
-        _input = prompt.format_prompt(question=prompt_template)
-        output = chat_model(_input.to_messages())
-
-    # Log token length if running with Gradio
-    try:
-        nt = num_tokens_from_string(_input.to_string(), "cl100k_base")
-        logger.info(f'Prompt token length --- {nt}')
-    except:
-        pass
-
-    # Parse the output
-    try:
-        # Check if output is of type 'ai' and parse accordingly
-        if output.type == 'ai':
-            parsed_content = output.content
-            logger.info(f'Formatted JSON\n{parsed_content}')
-        else:
-            # If not 'ai', log and set parsed_content to None or a default value
-            logger.error('Output type is not "ai". Unable to parse.')
-            return None
-
-        # Clean up the parsed content
-        parsed_content = parsed_content.replace('\n', "").replace('\t', "").replace('|', "")
-
-        # Attempt to parse the cleaned content
-        try:
-            refined_response = output_parser.parse(parsed_content)
-            return refined_response
-        except Exception as parse_error:
-            # Handle parsing errors specifically
-            logger.error(f'Parsing Error: {parse_error}')
-            return structured_output_parser_for_function_calls_fail(is_azure, MODEL, parsed_content, logger, llm, prompt_version, is_helper)
-
-    except Exception as e:
-        # Handle any other exceptions that might occur
-        logger.error(f'Unexpected Error: {e}')
-        return None
-
-def structured_output_parser_for_function_calls_fail(is_azure, MODEL, failed_response, logger, llm, prompt_version, is_helper=False, try_ind=0):
-    if try_ind == 0:
-        original_failed_response = failed_response
-    if try_ind > 5:
-        return None
-
-    # prompt_redo = PROMPT_JSON('helper' if is_helper else 'dict', failed_response)
-    Prompt = PromptCatalog()
-    if prompt_version in ['prompt_v1_verbose', 'prompt_v1_verbose_noDomainKnowledge']:
-        prompt_redo = Prompt.prompt_gpt_redo_v1(failed_response)
-    elif prompt_version in ['prompt_v2_json_rules']:
-        prompt_redo = Prompt.prompt_gpt_redo_v2(failed_response)
-    else:
-        prompt_redo = Prompt.prompt_v2_custom_redo(failed_response, is_palm=False)
-
-    response_schemas = [
-        ResponseSchema(name="Summary", description="A one sentence summary of the content"),
-        ResponseSchema(name="Dictionary", description='Formatted JSON object')
-    ]
-
-    output_parser = StructuredOutputParser.from_response_schemas(response_schemas)
-    format_instructions = output_parser.get_format_instructions()
-
-    prompt = ChatPromptTemplate(
-        messages=[
-            HumanMessagePromptTemplate.from_template("The following text contains JSON formatted text, but there is an error that you need to correct.\n{format_instructions}\n{question}")
-        ],
-        input_variables=["question"],
-        partial_variables={"format_instructions": format_instructions}
-    )
-
-    _input = prompt.format_prompt(question=prompt_redo)
-
-    # Log token length if running with Gradio
-    try:
-        nt = num_tokens_from_string(_input.to_string(), "cl100k_base")
-        logger.info(f'Prompt Redo token length --- {nt}')
-    except:
-        pass
-
-    if is_azure:
-        msg = HumanMessage(content=_input.to_string())
-        output = azure_call(llm, [msg])
-    else:
-        chat_model = ChatOpenAI(temperature=0, model=MODEL)
-        output = chat_model(_input.to_messages())
-
-    try:
-        refined_response = output_parser.parse(output.content)
-    except json.decoder.JSONDecodeError as e:
-        try_ind += 1
-        error_message = str(e)
-        redo_content = f'The error messsage is: {error_message}\nThe broken JSON object is: {original_failed_response}'  # Use original_failed_response here
-        logger.info(f'[Failed JSON Object]\n{original_failed_response}')  # And here
-        refined_response = structured_output_parser_for_function_calls_fail(
-            is_azure, MODEL, redo_content, logger, llm, prompt_version, is_helper, try_ind, original_failed_response
-        )
-    except:
-        try_ind += 1
-        logger.info(f'[Failed JSON Object]\n{original_failed_response}')  # And here
-        refined_response = structured_output_parser_for_function_calls_fail(
-            is_azure, MODEL, original_failed_response, logger, llm, prompt_version, is_helper, try_ind, original_failed_response
-        )
-
-    return refined_response
-
-
-
-
-class FunctionSchema:
-    def __init__(self):
-        pass
-
-    def format_C21_AA_V1(self):
-        return [
-            {
-                "name": "format_C21_AA_V1",
-                "description": "Format the given data into a specific dictionary",
-                "parameters": {
-                    "type": "object",
-                    "properties": {},  # specify parameters here if your function requires any
-                    "required": []  # list of required parameters
-                },
-                "output_type": "json",
-                "output_schema": {
-                    "type": "object",
-                    "properties": {
-                        "Dictionary": {
-                            "type": "object",
-                            "properties": {
-                                "Catalog Number": {"type": "array", "items": {"type": "string"}},
-                                "Genus": {"type": "array", "items": {"type": "string"}},
-                                "Species": {"type": "array", "items": {"type": "string"}},
-                                "subspecies": {"type": "array", "items": {"type": "string"}},
-                                "variety": {"type": "array", "items": {"type": "string"}},
-                                "forma": {"type": "array", "items": {"type": "string"}},
-                                "Country": {"type": "array", "items": {"type": "string"}},
-                                "State": {"type": "array", "items": {"type": "string"}},
-                                "County": {"type": "array", "items": {"type": "string"}},
-                                "Locality Name": {"type": "array", "items": {"type": "string"}},
-                                "Min Elevation": {"type": "array", "items": {"type": "string"}},
-                                "Max Elevation": {"type": "array", "items": {"type": "string"}},
-                                "Elevation Units": {"type": "array", "items": {"type": "string"}},
-                                "Verbatim Coordinates": {"type": "array", "items": {"type": "string"}},
-                                "Datum": {"type": "array", "items": {"type": "string"}},
-                                "Cultivated": {"type": "array", "items": {"type": "string"}},
-                                "Habitat": {"type": "array", "items": {"type": "string"}},
-                                "Collectors": {"type": "array", "items": {"type": "string"}},
-                                "Collector Number": {"type": "array", "items": {"type": "string"}},
-                                "Verbatim Date": {"type": "array", "items": {"type": "string"}},
-                                "Date": {"type": "array", "items": {"type": "string"}},
-                                "End Date": {"type": "array", "items": {"type": "string"}}
-                            }
-                        },
-                        "SpeciesName": {
-                            "type": "object",
-                            "properties": {
-                                "taxonomy": {"type": "array", "items": {"type": "string"}}
-                            }
-                        }
-                    }
-                }
-            }
-        ]
-
-    def format_C21_AA_V1_helper(self):
-        return [
-            {
-                "name": "format_C21_AA_V1_helper",
-                "description": "Helper function for format_C21_AA_V1 to further format the given data",
-                "parameters": {
-                    "type": "object",
-                    "properties": {},  # specify parameters here if your function requires any
-                    "required": []  # list of required parameters
-                },
-                "output_type": "json",
-                "output_schema": {
-                    "type": "object",
-                    "properties": {
-                        "Dictionary": {
-                            "type": "object",
-                            "properties": {
-                                "TAXONOMY": {
-                                    "type": "object",
-                                    "properties": {
-                                        "Order": {"type": "array", "items": {"type": "string"}},
-                                        "Family": {"type": "array", "items": {"type": "string"}},
-                                        "Genus":{"type": "array", "items": {"type": "string"}},
-                                        "Species": {"type": "array", "items": {"type": "string"}},
-                                        "Subspecies": {"type": "array", "items": {"type": "string"}},
-                                        "Variety": {"type": "array", "items": {"type": "string"}},
-                                        "Forma": {"type": "array", "items": {"type": "string"}},
-                                    }
-                                },
-                                "GEOGRAPHY": {
-                                    "type": "object",
-                                    "properties": {
-                                        "Country": {"type": "array", "items": {"type": "string"}},
-                                        "State": {"type": "array", "items": {"type": "string"}},
-                                        "Prefecture": {"type": "array", "items": {"type": "string"}},
-                                        "Province": {"type": "array", "items": {"type": "string"}},
-                                        "District": {"type": "array", "items": {"type": "string"}},
-                                        "County": {"type": "array", "items": {"type": "string"}},
-                                        "City": {"type": "array", "items": {"type": "string"}},
-                                        "Administrative Division": {"type": "array", "items": {"type": "string"}},
-                                    }
-                                },
-                                "LOCALITY": {
-                                    "type": "object",
-                                    "properties": {
-                                        "Landscape": {"type": "array", "items": {"type": "string"}},
-                                        "Nearby Places": {"type": "array", "items": {"type": "string"}},
-                                    }
-                                },
-                                "COLLECTING": {
-                                    "type": "object",
-                                    "properties": {
-                                        "Collector": {"type": "array", "items": {"type": "string"}},
-                                        "Collector's Number": {"type": "array", "items": {"type": "string"}},
-                                        "Verbatim Date": {"type": "array", "items": {"type": "string"}},
-                                        "Formatted Date": {"type": "array", "items": {"type": "string"}},
-                                        "Cultivation Status": {"type": "array", "items": {"type": "string"}},
-                                        "Habitat Description": {"type": "array", "items": {"type": "string"}},
-                                    }
-                                },
-                                "MISCELLANEOUS": {
-                                    "type": "object",
-                                    "properties": {
-                                        "Additional Information": {"type": "array", "items": {"type": "string"}},
-                                    }
-                                }
-                            }
-                        },
-                        "Summary": {
-                            "type": "object",
-                            "properties": {
-                                "Content Summary": {"type": "array", "items": {"type": "string"}}
-                            }
-                        }
-                    }
-                }
-            }
-        ]

vouchervision/LLM_local_MistralAI.py

@@ -0,0 +1,211 @@
+import json, torch, transformers, gc
+from transformers import BitsAndBytesConfig
+from langchain.output_parsers import RetryWithErrorOutputParser
+from langchain.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser
+from huggingface_hub import hf_hub_download
+from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
+
+from vouchervision.utils_LLM import SystemLoadMonitor, count_tokens
+from vouchervision.utils_LLM_JSON_validation import validate_and_align_JSON_keys_with_template
+from vouchervision.utils_taxonomy_WFO import validate_taxonomy_WFO
+from vouchervision.utils_geolocate_HERE import validate_coordinates_here
+
+'''
+https://python.langchain.com/docs/integrations/llms/huggingface_pipelines
+'''
+
+class LocalMistralHandler: 
+    RETRY_DELAY = 2  # Wait 2 seconds before retrying
+    MAX_RETRIES = 5  # Maximum number of retries
+    STARTING_TEMP = 0.1
+    TOKENIZER_NAME = None
+    VENDOR = 'mistral'
+    MAX_GPU_MONITORING_INTERVAL = 2  # seconds
+
+    def __init__(self, logger, model_name, JSON_dict_structure):
+        self.logger = logger
+        self.has_GPU = torch.cuda.is_available()
+        self.monitor = SystemLoadMonitor(logger)
+
+        self.model_name = model_name
+        self.model_id = f"mistralai/{self.model_name}"
+        name_parts = self.model_name.split('-')
+
+        self.model_path = hf_hub_download(repo_id=self.model_id, repo_type="model",filename="config.json")
+
+
+        self.JSON_dict_structure = JSON_dict_structure
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp 
+
+        system_prompt = "You are a helpful AI assistant who answers queries a JSON dictionary as specified by the user."
+        template = """
+            <s>[INST]{}[/INST]</s>
+
+            [INST]{}[/INST]
+            """.format(system_prompt, "{query}")
+
+        # Create a prompt from the template so we can use it with Langchain
+        self.prompt = PromptTemplate(template=template, input_variables=["query"])
+
+        # Set up a parser
+        self.parser = JsonOutputParser()
+
+        self._set_config()
+
+
+    # def _clear_VRAM(self):
+    #     # Clear CUDA cache if it's being used
+    #     if self.has_GPU:
+    #         self.local_model = None
+    #         self.local_model_pipeline = None
+    #         del self.local_model
+    #         del self.local_model_pipeline
+    #         gc.collect()  # Explicitly invoke garbage collector
+    #         torch.cuda.empty_cache()
+    #     else:
+    #         self.local_model_pipeline = None
+    #         self.local_model = None
+    #         del self.local_model_pipeline
+    #         del self.local_model
+    #         gc.collect()  # Explicitly invoke garbage collector
+
+
+    def _set_config(self):
+        # self._clear_VRAM()
+        self.config = {'max_new_tokens': 1024,
+                'temperature': self.starting_temp,
+                'seed': 2023,
+                'top_p': 1,
+                'top_k': 40,
+                'do_sample': True,
+                'n_ctx':4096,
+
+                # Activate 4-bit precision base model loading
+                'use_4bit': True,
+                # Compute dtype for 4-bit base models
+                'bnb_4bit_compute_dtype': "float16",
+                # Quantization type (fp4 or nf4)
+                'bnb_4bit_quant_type': "nf4",
+                # Activate nested quantization for 4-bit base models (double quantization)
+                'use_nested_quant': False,
+                }
+        
+        compute_dtype = getattr(torch,self.config.get('bnb_4bit_compute_dtype') )
+
+        self.bnb_config = BitsAndBytesConfig(
+            load_in_4bit=self.config.get('use_4bit'),
+            bnb_4bit_quant_type=self.config.get('bnb_4bit_quant_type'),
+            bnb_4bit_compute_dtype=compute_dtype,
+            bnb_4bit_use_double_quant=self.config.get('use_nested_quant'),
+        )
+
+        # Check GPU compatibility with bfloat16
+        if compute_dtype == torch.float16 and self.config.get('use_4bit'):
+            major, _ = torch.cuda.get_device_capability()
+            if major >= 8:
+                # print("=" * 80)
+                # print("Your GPU supports bfloat16: accelerate training with bf16=True")
+                # print("=" * 80)
+                self.b_float_opt =  torch.bfloat16
+
+            else:
+                self.b_float_opt =  torch.float16
+        self._build_model_chain_parser()
+    
+
+    def _adjust_config(self):
+        new_temp = self.adjust_temp + self.temp_increment
+        self.json_report.set_text(text_main=f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.logger.info(f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.adjust_temp += self.temp_increment
+
+    
+    def _reset_config(self):
+        self.json_report.set_text(text_main=f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.logger.info(f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.adjust_temp = self.starting_temp
+    
+
+    def _build_model_chain_parser(self):
+        self.local_model_pipeline = transformers.pipeline("text-generation", 
+                    model=self.model_id,
+                    max_new_tokens=self.config.get('max_new_tokens'),
+                    top_k=self.config.get('top_k'),
+                    top_p=self.config.get('top_p'),
+                    do_sample=self.config.get('do_sample'),
+                    model_kwargs={"torch_dtype": self.b_float_opt, 
+                                "load_in_4bit": True, 
+                                "quantization_config": self.bnb_config})
+        self.local_model = HuggingFacePipeline(pipeline=self.local_model_pipeline)
+        # Set up the retry parser with the runnable
+        self.retry_parser = RetryWithErrorOutputParser.from_llm(parser=self.parser, llm=self.local_model, max_retries=self.MAX_RETRIES)
+        # Create an llm chain with LLM and prompt
+        self.chain = self.prompt | self.local_model  # LCEL
+
+
+    def call_llm_local_MistralAI(self, prompt_template, json_report):
+        self.json_report = json_report
+        self.json_report.set_text(text_main=f'Sending request to {self.model_name}')
+        self.monitor.start_monitoring_usage()
+        
+        nt_in = 0
+        nt_out = 0
+
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            try:
+                # Dynamically set the temperature for this specific request
+                model_kwargs = {"temperature": self.adjust_temp}
+                
+                # Invoke the chain to generate prompt text
+                results = self.chain.invoke({"query": prompt_template, "model_kwargs": model_kwargs})
+
+                # Use retry_parser to parse the response with retry logic
+                output = self.retry_parser.parse_with_prompt(results, prompt_value=prompt_template)
+
+                if output is None:
+                    self.logger.error(f'Failed to extract JSON from:\n{results}')
+                    self._adjust_config()
+                    del results
+
+                else:
+                    nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                    nt_out = count_tokens(results, self.VENDOR, self.TOKENIZER_NAME)
+
+                    output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                    
+                    if output is None:
+                        self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{results}')
+                        self._adjust_config()
+                    else:
+                        json_report.set_text(text_main=f'Working on WFO and Geolocation')
+                        output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False) # Make configurable if needed
+                        output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False) # Make configurable if needed
+
+                        self.logger.info(f"Formatted JSON:\n{json.dumps(output,indent=4)}")
+
+                        self.monitor.stop_monitoring_report_usage()                
+
+                        if self.adjust_temp != self.starting_temp:
+                            self._reset_config()
+                        
+                        json_report.set_text(text_main=f'LLM call successful')
+                        del results
+                        return output, nt_in, nt_out, WFO_record, GEO_record
+            except Exception as e:
+                self.logger.error(f'{e}')
+                self._adjust_config()           
+                
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        self.json_report.set_text(text_main=f'Failed to extract valid JSON after [{ind}] attempts')
+
+        self.monitor.stop_monitoring_report_usage()                
+        json_report.set_text(text_main=f'LLM call failed')
+
+        self._reset_config()
+        return None, nt_in, nt_out, None, None
+

vouchervision/LLM_local_MistralAI_batch.py

@@ -0,0 +1,256 @@
+import json, torch, transformers, gc
+from transformers import BitsAndBytesConfig
+from langchain.output_parsers import RetryWithErrorOutputParser
+from langchain.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser
+from huggingface_hub import hf_hub_download
+from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
+
+from utils_LLM import validate_and_align_JSON_keys_with_template, count_tokens, validate_taxonomy_WFO, validate_coordinates_here, remove_colons_and_double_apostrophes, SystemLoadMonitor
+
+'''
+https://python.langchain.com/docs/integrations/llms/huggingface_pipelines
+'''
+
+from torch.utils.data import Dataset, DataLoader
+# Dataset for handling prompts
+class PromptDataset(Dataset):
+    def __init__(self, prompts):
+        self.prompts = prompts
+
+    def __len__(self):
+        return len(self.prompts)
+
+    def __getitem__(self, idx):
+        return self.prompts[idx]
+    
+class LocalMistralHandler: 
+    RETRY_DELAY = 2  # Wait 2 seconds before retrying
+    MAX_RETRIES = 5  # Maximum number of retries
+    STARTING_TEMP = 0.1
+    TOKENIZER_NAME = None
+    VENDOR = 'mistral'
+    MAX_GPU_MONITORING_INTERVAL = 2  # seconds
+
+    def __init__(self, logger, model_name, JSON_dict_structure):
+        self.logger = logger
+        self.has_GPU = torch.cuda.is_available()
+        self.monitor = SystemLoadMonitor(logger)
+
+        self.model_name = model_name
+        self.model_id = f"mistralai/{self.model_name}"
+        name_parts = self.model_name.split('-')
+
+        self.model_path = hf_hub_download(repo_id=self.model_id, repo_type="model",filename="config.json")
+
+
+        self.JSON_dict_structure = JSON_dict_structure
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp 
+
+        system_prompt = "You are a helpful AI assistant who answers queries a JSON dictionary as specified by the user."
+        template = """
+            <s>[INST]{}[/INST]</s>
+
+            [INST]{}[/INST]
+            """.format(system_prompt, "{query}")
+
+        # Create a prompt from the template so we can use it with Langchain
+        self.prompt = PromptTemplate(template=template, input_variables=["query"])
+
+        # Set up a parser
+        self.parser = JsonOutputParser()
+
+        self._set_config()
+
+
+    def _clear_VRAM(self):
+        # Clear CUDA cache if it's being used
+        if self.has_GPU:
+            self.local_model = None
+            self.local_model_pipeline = None
+            del self.local_model
+            del self.local_model_pipeline
+            gc.collect()  # Explicitly invoke garbage collector
+            torch.cuda.empty_cache()
+        else:
+            self.local_model_pipeline = None
+            self.local_model = None
+            del self.local_model_pipeline
+            del self.local_model
+            gc.collect()  # Explicitly invoke garbage collector
+
+
+    def _set_config(self):
+        self._clear_VRAM()
+        self.config = {'max_new_tokens': 1024,
+                'temperature': self.starting_temp,
+                'seed': 2023,
+                'top_p': 1,
+                'top_k': 40,
+                'do_sample': True,
+                'n_ctx':4096,
+
+                # Activate 4-bit precision base model loading
+                'use_4bit': True,
+                # Compute dtype for 4-bit base models
+                'bnb_4bit_compute_dtype': "float16",
+                # Quantization type (fp4 or nf4)
+                'bnb_4bit_quant_type': "nf4",
+                # Activate nested quantization for 4-bit base models (double quantization)
+                'use_nested_quant': False,
+                }
+        
+        compute_dtype = getattr(torch,self.config.get('bnb_4bit_compute_dtype') )
+
+        self.bnb_config = BitsAndBytesConfig(
+            load_in_4bit=self.config.get('use_4bit'),
+            bnb_4bit_quant_type=self.config.get('bnb_4bit_quant_type'),
+            bnb_4bit_compute_dtype=compute_dtype,
+            bnb_4bit_use_double_quant=self.config.get('use_nested_quant'),
+        )
+
+        # Check GPU compatibility with bfloat16
+        if compute_dtype == torch.float16 and self.config.get('use_4bit'):
+            major, _ = torch.cuda.get_device_capability()
+            if major >= 8:
+                # print("=" * 80)
+                # print("Your GPU supports bfloat16: accelerate training with bf16=True")
+                # print("=" * 80)
+                self.b_float_opt =  torch.bfloat16
+
+            else:
+                self.b_float_opt =  torch.float16
+        self._build_model_chain_parser()
+    
+
+    def _adjust_config(self):
+        self.logger.info(f'Incrementing temperature and reloading model')
+        self._clear_VRAM()
+        self.adjust_temp += self.temp_increment
+        self.config['temperature'] = self.adjust_temp
+        self._build_model_chain_parser()
+    
+
+    def _build_model_chain_parser(self):
+        self.local_model_pipeline = transformers.pipeline("text-generation", 
+                        model=self.model_id,
+                        max_new_tokens=self.config.get('max_new_tokens'),
+                        temperature=self.config.get('temperature'),
+                        top_k=self.config.get('top_k'),
+                        top_p=self.config.get('top_p'),
+                        do_sample=self.config.get('do_sample'),
+                        model_kwargs={"torch_dtype": self.b_float_opt, 
+                                    "load_in_4bit": True, 
+                                    "quantization_config": self.bnb_config})
+        self.local_model = HuggingFacePipeline(pipeline=self.local_model_pipeline)
+        # Set up the retry parser with the runnable
+        self.retry_parser = RetryWithErrorOutputParser.from_llm(parser=self.parser, llm=self.local_model, max_retries=self.MAX_RETRIES)
+        # Create an llm chain with LLM and prompt
+        self.chain = self.prompt | self.local_model  # LCEL
+
+    '''
+    def call_llm_local_MistralAI(self, prompt_template):
+        self.monitor.start_monitoring_usage()
+        
+        nt_in = 0
+        nt_out = 0
+
+        ind = 0
+        while (ind < self.MAX_RETRIES):
+            ind += 1
+            # Invoke the chain to generate prompt text
+            results = self.chain.invoke({"query": prompt_template})
+
+            # Use retry_parser to parse the response with retry logic
+            output = self.retry_parser.parse_with_prompt(results, prompt_value=prompt_template)
+
+            if output is None:
+                self.logger.error(f'Failed to extract JSON from:\n{results}')
+                self._adjust_config()
+                del results
+
+            else:
+                nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                nt_out = count_tokens(results, self.VENDOR, self.TOKENIZER_NAME)
+
+                output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False) # Make configurable if needed
+                output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False) # Make configurable if needed
+
+                self.logger.info(f"Formatted JSON:\n{json.dumps(output,indent=4)}")
+
+                self.monitor.stop_monitoring_report_usage()                
+
+                if self.adjust_temp != self.starting_temp:
+                    self._set_config()
+                
+                del results
+                return output, nt_in, nt_out, WFO_record, GEO_record
+        
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+
+        self.monitor.stop_monitoring_report_usage()                
+
+        self._set_config()
+        return None, nt_in, nt_out, None, None
+    '''
+    def call_llm_local_MistralAI(self, prompts, batch_size=4):
+        self.monitor.start_monitoring_usage()
+
+        dataset = PromptDataset(prompts)
+        data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=False)
+
+        all_results = []
+        for batch_prompts in data_loader:
+            batch_results = self._process_batch(batch_prompts)
+            all_results.extend(batch_results)
+
+        self.monitor.stop_monitoring_report_usage()
+
+        if self.adjust_temp != self.starting_temp:
+            self._set_config()
+
+        return all_results
+
+    def _process_batch(self, batch_prompts):
+        batch_results = []
+        for prompt in batch_prompts:
+            output, nt_in, nt_out, WFO_record, GEO_record = self._process_single_prompt(prompt)
+            if output is not None:
+                batch_results.append({
+                    "output": output,
+                    "nt_in": nt_in,
+                    "nt_out": nt_out,
+                    "WFO_record": WFO_record,
+                    "GEO_record": GEO_record
+                })
+        return batch_results
+
+    def _process_single_prompt(self, prompt_template):
+        nt_in = nt_out = 0
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            results = self.chain.invoke({"query": prompt_template})
+            output = self.retry_parser.parse_with_prompt(results, prompt_value=prompt_template)
+
+            if output is None:
+                self.logger.error(f'Failed to extract JSON from:\n{results}')
+                self._adjust_config()
+                del results
+            else:
+                nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                nt_out = count_tokens(results, self.VENDOR, self.TOKENIZER_NAME)
+
+                output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False)
+                output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False)
+
+                self.logger.info(f"Formatted JSON:\n{json.dumps(output, indent=4)}")
+                del results
+                return output, nt_in, nt_out, WFO_record, GEO_record
+
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        return None, nt_in, nt_out, None, None

vouchervision/LLM_local_MistralAI_batch_async.py

@@ -0,0 +1,210 @@
+import json, torch, transformers, gc
+from transformers import BitsAndBytesConfig
+from langchain.output_parsers import RetryWithErrorOutputParser
+from langchain.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser
+from huggingface_hub import hf_hub_download
+from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
+import asyncio
+
+from utils_LLM import validate_and_align_JSON_keys_with_template, count_tokens, validate_taxonomy_WFO, validate_coordinates_here, remove_colons_and_double_apostrophes, SystemLoadMonitor
+
+'''
+https://python.langchain.com/docs/integrations/llms/huggingface_pipelines
+'''
+
+from torch.utils.data import Dataset, DataLoader
+# Dataset for handling prompts
+class PromptDataset(Dataset):
+    def __init__(self, prompts):
+        self.prompts = prompts
+
+    def __len__(self):
+        return len(self.prompts)
+
+    def __getitem__(self, idx):
+        return self.prompts[idx]
+    
+class LocalMistralHandler: 
+    RETRY_DELAY = 2  # Wait 2 seconds before retrying
+    MAX_RETRIES = 5  # Maximum number of retries
+    STARTING_TEMP = 0.1
+    TOKENIZER_NAME = None
+    VENDOR = 'mistral'
+    MAX_GPU_MONITORING_INTERVAL = 2  # seconds
+
+    def __init__(self, logger, model_name, JSON_dict_structure):
+        self.logger = logger
+        self.has_GPU = torch.cuda.is_available()
+        self.monitor = SystemLoadMonitor(logger)
+
+        self.model_name = model_name
+        self.model_id = f"mistralai/{self.model_name}"
+        name_parts = self.model_name.split('-')
+
+        self.model_path = hf_hub_download(repo_id=self.model_id, repo_type="model",filename="config.json")
+
+
+        self.JSON_dict_structure = JSON_dict_structure
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp 
+
+        system_prompt = "You are a helpful AI assistant who answers queries a JSON dictionary as specified by the user."
+        template = """
+            <s>[INST]{}[/INST]</s>
+
+            [INST]{}[/INST]
+            """.format(system_prompt, "{query}")
+
+        # Create a prompt from the template so we can use it with Langchain
+        self.prompt = PromptTemplate(template=template, input_variables=["query"])
+
+        # Set up a parser
+        self.parser = JsonOutputParser()
+
+        self._set_config()
+
+
+    def _clear_VRAM(self):
+        # Clear CUDA cache if it's being used
+        if self.has_GPU:
+            self.local_model = None
+            self.local_model_pipeline = None
+            del self.local_model
+            del self.local_model_pipeline
+            gc.collect()  # Explicitly invoke garbage collector
+            torch.cuda.empty_cache()
+        else:
+            self.local_model_pipeline = None
+            self.local_model = None
+            del self.local_model_pipeline
+            del self.local_model
+            gc.collect()  # Explicitly invoke garbage collector
+
+
+    def _set_config(self):
+        self._clear_VRAM()
+        self.config = {'max_new_tokens': 1024,
+                'temperature': self.starting_temp,
+                'seed': 2023,
+                'top_p': 1,
+                'top_k': 40,
+                'do_sample': True,
+                'n_ctx':4096,
+
+                # Activate 4-bit precision base model loading
+                'use_4bit': True,
+                # Compute dtype for 4-bit base models
+                'bnb_4bit_compute_dtype': "float16",
+                # Quantization type (fp4 or nf4)
+                'bnb_4bit_quant_type': "nf4",
+                # Activate nested quantization for 4-bit base models (double quantization)
+                'use_nested_quant': False,
+                }
+        
+        compute_dtype = getattr(torch,self.config.get('bnb_4bit_compute_dtype') )
+
+        self.bnb_config = BitsAndBytesConfig(
+            load_in_4bit=self.config.get('use_4bit'),
+            bnb_4bit_quant_type=self.config.get('bnb_4bit_quant_type'),
+            bnb_4bit_compute_dtype=compute_dtype,
+            bnb_4bit_use_double_quant=self.config.get('use_nested_quant'),
+        )
+
+        # Check GPU compatibility with bfloat16
+        if compute_dtype == torch.float16 and self.config.get('use_4bit'):
+            major, _ = torch.cuda.get_device_capability()
+            if major >= 8:
+                # print("=" * 80)
+                # print("Your GPU supports bfloat16: accelerate training with bf16=True")
+                # print("=" * 80)
+                self.b_float_opt =  torch.bfloat16
+
+            else:
+                self.b_float_opt =  torch.float16
+        self._build_model_chain_parser()
+    
+
+    def _adjust_config(self):
+        self.logger.info(f'Incrementing temperature and reloading model')
+        self._clear_VRAM()
+        self.adjust_temp += self.temp_increment
+        self.config['temperature'] = self.adjust_temp
+        self._build_model_chain_parser()
+    
+
+    def _build_model_chain_parser(self):
+        self.local_model_pipeline = transformers.pipeline("text-generation", 
+                        model=self.model_id,
+                        max_new_tokens=self.config.get('max_new_tokens'),
+                        temperature=self.config.get('temperature'),
+                        top_k=self.config.get('top_k'),
+                        top_p=self.config.get('top_p'),
+                        do_sample=self.config.get('do_sample'),
+                        model_kwargs={"torch_dtype": self.b_float_opt, 
+                                    "load_in_4bit": True, 
+                                    "quantization_config": self.bnb_config})
+        self.local_model = HuggingFacePipeline(pipeline=self.local_model_pipeline)
+        # Set up the retry parser with the runnable
+        self.retry_parser = RetryWithErrorOutputParser.from_llm(parser=self.parser, llm=self.local_model, max_retries=self.MAX_RETRIES)
+        # Create an llm chain with LLM and prompt
+        self.chain = self.prompt | self.local_model  
+
+    def call_llm_local_MistralAI(self, prompts, batch_size=2):
+        # Wrap the async call with asyncio.run
+
+        dataset = PromptDataset(prompts)
+        data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=False)
+
+        all_results = asyncio.run(self._process_all_batches(data_loader))
+
+
+        if self.adjust_temp != self.starting_temp:
+            self._set_config()
+
+        return all_results
+    
+    async def _process_batch(self, batch_prompts):
+        # Create and manage async tasks for each prompt in the batch
+        tasks = [self._process_single_prompt(prompt) for prompt in batch_prompts]
+        return await asyncio.gather(*tasks)
+
+    async def _process_all_batches(self, data_loader):
+        # Process all batches asynchronously
+        results = []
+        for batch_prompts in data_loader:
+            batch_results = await self._process_batch(batch_prompts)
+            results.extend(batch_results)
+        return results
+
+    async def _process_single_prompt(self, prompt_template):
+        self.monitor.start_monitoring_usage()
+        nt_in = nt_out = 0
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            results = self.chain.invoke({"query": prompt_template})
+            output = self.retry_parser.parse_with_prompt(results, prompt_value=prompt_template)
+
+            if output is None:
+                self.logger.error(f'Failed to extract JSON from:\n{results}')
+                self._adjust_config()
+                del results
+            else:
+                nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                nt_out = count_tokens(results, self.VENDOR, self.TOKENIZER_NAME)
+
+                output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False)
+                output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False)
+
+                self.logger.info(f"Formatted JSON:\n{json.dumps(output, indent=4)}")
+                del results
+                self.monitor.stop_monitoring_report_usage()
+
+                return output, nt_in, nt_out, WFO_record, GEO_record
+        self.monitor.stop_monitoring_report_usage()
+
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        return None, nt_in, nt_out, None, None

vouchervision/LLM_local_cpu_MistralAI.py

@@ -0,0 +1,205 @@
+import os, json, gc
+import time
+import torch
+import transformers
+import random
+from transformers import BitsAndBytesConfig#, AutoModelForCausalLM, AutoTokenizer
+from langchain.output_parsers import RetryWithErrorOutputParser
+from langchain.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser
+from langchain_experimental.llms import JsonFormer
+from langchain.tools import tool
+# from langchain_community.llms import CTransformers
+# from ctransformers import AutoModelForCausalLM, AutoConfig, Config
+
+from langchain_community.llms import LlamaCpp
+# from langchain.callbacks.manager import CallbackManager
+from langchain.callbacks.base import BaseCallbackHandler
+from huggingface_hub import hf_hub_download
+
+
+from vouchervision.utils_LLM import SystemLoadMonitor, count_tokens
+from vouchervision.utils_LLM_JSON_validation import validate_and_align_JSON_keys_with_template
+from vouchervision.utils_taxonomy_WFO import validate_taxonomy_WFO
+from vouchervision.utils_geolocate_HERE import validate_coordinates_here
+
+class LocalCPUMistralHandler: 
+    RETRY_DELAY = 2  # Wait 2 seconds before retrying
+    MAX_RETRIES = 5  # Maximum number of retries
+    STARTING_TEMP = 0.1
+    TOKENIZER_NAME = None
+    VENDOR = 'mistral'
+    SEED = 2023
+
+
+    def __init__(self, logger, model_name, JSON_dict_structure):
+        self.logger = logger
+        self.monitor = SystemLoadMonitor(logger)
+        self.has_GPU = torch.cuda.is_available()
+        self.JSON_dict_structure = JSON_dict_structure
+
+        self.model_file = None
+        self.model_name = model_name
+
+        # https://medium.com/@scholarly360/mistral-7b-complete-guide-on-colab-129fa5e9a04d
+        self.model_name = "Mistral-7B-Instruct-v0.2-GGUF"  #huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GGUF mistral-7b-instruct-v0.2.Q4_K_M.gguf --local-dir /home/brlab/.cache --local-dir-use-symlinks False
+        self.model_id = f"TheBloke/{self.model_name}"
+        name_parts = self.model_name.split('-')
+        
+        if self.model_name == "Mistral-7B-Instruct-v0.2-GGUF":
+            self.model_file = 'mistral-7b-instruct-v0.2.Q4_K_M.gguf'
+            self.model_path = hf_hub_download(repo_id=self.model_id,
+                                 filename=self.model_file,
+                                 repo_type="model")
+        else:
+            raise f"Unsupported GGUF model name"
+
+        # self.model_id = f"mistralai/{self.model_name}"
+        self.gpu_usage = {'max_load': 0, 'max_memory_usage': 0, 'monitoring': True}
+
+        self.starting_temp = float(self.STARTING_TEMP)
+        self.temp_increment = float(0.2)
+        self.adjust_temp = self.starting_temp 
+
+        system_prompt = "You are a helpful AI assistant who answers queries with JSON objects and no explanations."
+        template = """
+            <s>[INST]{}[/INST]</s>
+
+            [INST]{}[/INST]
+            """.format(system_prompt, "{query}")
+
+        # Create a prompt from the template so we can use it with Langchain
+        self.prompt = PromptTemplate(template=template, input_variables=["query"])
+
+        # Set up a parser
+        self.parser = JsonOutputParser()
+
+        self._set_config()
+
+
+    # def _clear_VRAM(self):
+    #     # Clear CUDA cache if it's being used
+    #     if self.has_GPU:
+    #         self.local_model = None
+    #         del self.local_model
+    #         gc.collect()  # Explicitly invoke garbage collector
+    #         torch.cuda.empty_cache()
+    #     else:
+    #         self.local_model = None
+    #         del self.local_model
+    #         gc.collect()  # Explicitly invoke garbage collector
+
+
+    def _set_config(self): 
+        # self._clear_VRAM()
+        self.config = {'max_new_tokens': 1024,
+                'temperature': self.starting_temp,
+                'seed': self.SEED,
+                'top_p': 1,
+                'top_k': 40,
+                'n_ctx': 4096,
+                'do_sample': True,
+                }       
+        self._build_model_chain_parser()
+        
+
+    def _adjust_config(self):
+        new_temp = self.adjust_temp + self.temp_increment
+        self.json_report.set_text(text_main=f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.logger.info(f'Incrementing temperature from {self.adjust_temp} to {new_temp}')
+        self.adjust_temp += self.temp_increment
+        self.config['temperature'] = self.adjust_temp      
+
+    def _reset_config(self):
+        self.json_report.set_text(text_main=f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.logger.info(f'Resetting temperature from {self.adjust_temp} to {self.starting_temp}')
+        self.adjust_temp = self.starting_temp
+        self.config['temperature'] = self.starting_temp        
+        
+
+    def _build_model_chain_parser(self):
+        self.local_model = LlamaCpp(
+            model_path=self.model_path,
+            max_tokens=self.config.get('max_new_tokens'),
+            top_p=self.config.get('top_p'),
+            # callback_manager=callback_manager,
+            # n_gpu_layers=1,
+            # n_batch=512,
+            n_ctx=self.config.get('n_ctx'),
+            stop=["[INST]"],
+            verbose=False,
+            streaming=False,
+            )
+        # Set up the retry parser with the runnable
+        self.retry_parser = RetryWithErrorOutputParser.from_llm(parser=self.parser, llm=self.local_model, max_retries=self.MAX_RETRIES)
+        # Create an llm chain with LLM and prompt
+        self.chain = self.prompt | self.local_model
+
+
+    def call_llm_local_cpu_MistralAI(self, prompt_template, json_report):
+        self.json_report = json_report
+        self.json_report.set_text(text_main=f'Sending request to {self.model_name}')
+        self.monitor.start_monitoring_usage()
+
+        nt_in = 0
+        nt_out = 0
+
+        ind = 0
+        while ind < self.MAX_RETRIES:
+            ind += 1
+            try:
+                ### BELOW IS BASIC MISTRAL CALL
+                # mistral_prompt = f"<s>[INST] {prompt_template} [/INST]"
+                # results = self.local_model(mistral_prompt, temperature = 0.7, 
+                #             repetition_penalty = 1.15,
+                #             max_new_tokens = 2048)
+                # print(results)
+
+                model_kwargs = {"temperature": self.adjust_temp}
+                
+                # Invoke the chain to generate prompt text
+                results = self.chain.invoke({"query": prompt_template, "model_kwargs": model_kwargs})
+
+                # Use retry_parser to parse the response with retry logic
+                output = self.retry_parser.parse_with_prompt(results, prompt_value=prompt_template)
+
+                if output is None:
+                    self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{results}')
+                    self._adjust_config()
+
+                else:
+                    nt_in = count_tokens(prompt_template, self.VENDOR, self.TOKENIZER_NAME)
+                    nt_out = count_tokens(results, self.VENDOR, self.TOKENIZER_NAME)
+
+                    output = validate_and_align_JSON_keys_with_template(output, self.JSON_dict_structure)
+                    if output is None:
+                        self.logger.error(f'[Attempt {ind}] Failed to extract JSON from:\n{results}')
+                        self._adjust_config()
+                    else:
+                        json_report.set_text(text_main=f'Working on WFO and Geolocation')
+                        output, WFO_record = validate_taxonomy_WFO(output, replace_if_success_wfo=False) # Make configurable if needed
+                        output, GEO_record = validate_coordinates_here(output, replace_if_success_geo=False) # Make configurable if needed
+
+                        self.logger.info(f"Formatted JSON:\n{json.dumps(output,indent=4)}")
+
+                        self.monitor.stop_monitoring_report_usage()                
+
+                        if self.adjust_temp != self.starting_temp:
+                            self._reset_config()
+                        json_report.set_text(text_main=f'LLM call successful')
+                        return output, nt_in, nt_out, WFO_record, GEO_record
+                    
+            except Exception as e:
+                self.logger.error(f'{e}')
+                self._adjust_config()  
+        
+        self.logger.info(f"Failed to extract valid JSON after [{ind}] attempts")
+        self.json_report.set_text(text_main=f'Failed to extract valid JSON after [{ind}] attempts')
+
+        self.monitor.stop_monitoring_report_usage()                
+        self._reset_config()
+
+        json_report.set_text(text_main=f'LLM call failed')
+        return None, nt_in, nt_out, None, None
+
+

vouchervision/LM2_logger.py

@@ -1,9 +1,20 @@
 import logging, os, psutil, torch, platform, cpuinfo, yaml #py-cpuinfo
 from vouchervision.general_utils import get_datetime, print_main_warn, print_main_info
 
+class SanitizingFileHandler(logging.FileHandler):
+    def __init__(self, filename, mode='a', encoding=None, delay=False):
+        super().__init__(filename, mode, encoding, delay)
+
+    def emit(self, record):
+        try:
+            record.msg = record.msg.encode('utf-8', 'replace').decode('utf-8')
+        except Exception as e:
+            record.msg = f'[Error encoding text: {e}]'
+        super().emit(record)
+
 def start_logging(Dirs, cfg):
     run_name = cfg['leafmachine']['project']['run_name']
-    path_log = os.path.join(Dirs.path_log, '__'.join(['LM2-log',str(get_datetime()), run_name])+'.log')
+    path_log = os.path.join(Dirs.path_log, '__'.join(['LM2-log', str(get_datetime()), run_name]) + '.log')
 
     # Disable default StreamHandler
     logging.getLogger().handlers = []
@@ -12,9 +23,9 @@ def start_logging(Dirs, cfg):
     logger = logging.getLogger('Hardware Components')
     logger.setLevel(logging.DEBUG)
 
-    # create file handler and set level to debug
-    fh = logging.FileHandler(path_log)
-    fh.setLevel(logging.DEBUG)
+    # create custom sanitizing file handler and set level to debug
+    sanitizing_fh = SanitizingFileHandler(path_log, encoding='utf-8')
+    sanitizing_fh.setLevel(logging.DEBUG)
 
     # create console handler and set level to debug
     ch = logging.StreamHandler()
@@ -24,11 +35,11 @@ def start_logging(Dirs, cfg):
     formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s - %(message)s')
 
     # add formatter to handlers
-    fh.setFormatter(formatter)
+    sanitizing_fh.setFormatter(formatter)
     ch.setFormatter(formatter)
 
     # add handlers to logger
-    logger.addHandler(fh)
+    logger.addHandler(sanitizing_fh)
     logger.addHandler(ch)
 
     # Create a logger for the file handler

vouchervision/OCR_google_cloud_vision.py

@@ -1,12 +1,576 @@
-import os, io, sys, inspect
-from google.cloud import vision, storage
-from PIL import Image, ImageDraw
+import os, io, sys, inspect, statistics
+from statistics import mean 
+# from google.cloud import vision, storage
+from google.cloud import vision
+from google.cloud import vision_v1p3beta1 as vision_beta
+from PIL import Image, ImageDraw, ImageFont
+import colorsys
+from tqdm import tqdm
 
 currentdir = os.path.dirname(os.path.abspath(
     inspect.getfile(inspect.currentframe())))
 parentdir = os.path.dirname(currentdir)
 sys.path.append(parentdir)
 
+
+'''
+@misc{li2021trocr,
+      title={TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models}, 
+      author={Minghao Li and Tengchao Lv and Lei Cui and Yijuan Lu and Dinei Florencio and Cha Zhang and Zhoujun Li and Furu Wei},
+      year={2021},
+      eprint={2109.10282},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+'''
+
+class OCRGoogle:
+
+    BBOX_COLOR = "black"
+
+    def __init__(self, path, cfg, trOCR_model_version, trOCR_model, trOCR_processor, device):
+        self.path = path
+        self.cfg = cfg
+        self.do_use_trOCR = self.cfg['leafmachine']['project']['do_use_trOCR']
+        self.OCR_option = self.cfg['leafmachine']['project']['OCR_option']
+
+        # Initialize TrOCR components
+        self.trOCR_model_version = trOCR_model_version
+        self.trOCR_processor = trOCR_processor
+        self.trOCR_model = trOCR_model
+        self.device = device
+
+        self.hand_cleaned_text = None
+        self.hand_organized_text = None
+        self.hand_bounds = None
+        self.hand_bounds_word = None
+        self.hand_bounds_flat = None
+        self.hand_text_to_box_mapping = None
+        self.hand_height = None
+        self.hand_confidences = None
+        self.hand_characters = None
+
+        self.normal_cleaned_text = None
+        self.normal_organized_text = None
+        self.normal_bounds = None
+        self.normal_bounds_word = None
+        self.normal_text_to_box_mapping = None
+        self.normal_bounds_flat = None
+        self.normal_height = None
+        self.normal_confidences = None
+        self.normal_characters = None
+
+        self.trOCR_texts = None
+        self.trOCR_text_to_box_mapping = None
+        self.trOCR_bounds_flat = None
+        self.trOCR_height = None
+        self.trOCR_confidences = None
+        self.trOCR_characters = None
+
+    def detect_text_with_trOCR_using_google_bboxes(self, do_use_trOCR, logger):
+        CONFIDENCES = 0.80
+        MAX_NEW_TOKENS = 50
+
+        self.OCR_JSON_to_file = {}
+        
+        if not do_use_trOCR:
+            if self.OCR_option in ['normal',]:
+                self.OCR_JSON_to_file['OCR_printed'] = self.normal_organized_text
+                logger.info(f"Google_OCR_Standard:\n{self.normal_organized_text}")
+                return f"Google_OCR_Standard:\n{self.normal_organized_text}"
+            
+            if self.OCR_option in ['hand',]:
+                self.OCR_JSON_to_file['OCR_handwritten'] = self.hand_organized_text
+                logger.info(f"Google_OCR_Handwriting:\n{self.hand_organized_text}")
+                return f"Google_OCR_Handwriting:\n{self.hand_organized_text}"
+            
+            if self.OCR_option in ['both',]:
+                logger.info(f"Google_OCR_Standard:\n{self.normal_organized_text}\n\nGoogle_OCR_Handwriting:\n{self.hand_organized_text}")
+                return f"Google_OCR_Standard:\n{self.normal_organized_text}\n\nGoogle_OCR_Handwriting:\n{self.hand_organized_text}"
+
+        else:
+            logger.info(f'Supplementing with trOCR')
+
+            self.trOCR_texts = []
+            original_image = Image.open(self.path).convert("RGB")
+
+            if self.OCR_option in ['normal',]:
+                available_bounds = self.normal_bounds_word
+            elif self.OCR_option in ['hand',]:
+                available_bounds = self.hand_bounds_word 
+            elif self.OCR_option in ['both',]:
+                available_bounds = self.hand_bounds_word 
+            else:
+                raise
+
+            text_to_box_mapping = []
+            characters = []
+            height = []
+            confidences = []
+            for bound in tqdm(available_bounds, desc="Processing words using Google Vision bboxes"):
+                vertices = bound["vertices"]
+                
+
+                left = min([v["x"] for v in vertices])
+                top = min([v["y"] for v in vertices])
+                right = max([v["x"] for v in vertices])
+                bottom = max([v["y"] for v in vertices])
+
+                # Crop image based on Google's bounding box
+                cropped_image = original_image.crop((left, top, right, bottom))
+                pixel_values = self.trOCR_processor(cropped_image, return_tensors="pt").pixel_values
+
+                # Move pixel values to the appropriate device
+                pixel_values = pixel_values.to(self.device)
+
+                generated_ids = self.trOCR_model.generate(pixel_values, max_new_tokens=MAX_NEW_TOKENS)
+                extracted_text = self.trOCR_processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
+                self.trOCR_texts.append(extracted_text)
+
+                # For plotting 
+                word_length = max(vertex.get('x') for vertex in vertices) - min(vertex.get('x') for vertex in vertices)
+                num_symbols = len(extracted_text)
+
+                Yw = max(vertex.get('y') for vertex in vertices)
+                Yo = Yw - min(vertex.get('y') for vertex in vertices)
+                X = word_length / num_symbols if num_symbols > 0 else 0
+                H = int(X+(Yo*0.1))
+                height.append(H)
+
+                map_dict = {
+                    "vertices": vertices,
+                    "text": extracted_text  # Use the text extracted by trOCR
+                    }
+                text_to_box_mapping.append(map_dict)
+
+                characters.append(extracted_text)
+                confidences.append(CONFIDENCES)
+            
+            median_height = statistics.median(height) if height else 0
+            median_heights = [median_height * 1.5] * len(characters)
+
+            self.trOCR_texts = ' '.join(self.trOCR_texts)
+
+            self.trOCR_text_to_box_mapping = text_to_box_mapping
+            self.trOCR_bounds_flat = available_bounds
+            self.trOCR_height = median_heights
+            self.trOCR_confidences = confidences
+            self.trOCR_characters = characters
+
+            if self.OCR_option in ['normal',]:
+                self.OCR_JSON_to_file['OCR_printed'] = self.normal_organized_text
+                self.OCR_JSON_to_file['OCR_trOCR'] = self.trOCR_texts
+                logger.info(f"Google_OCR_Standard:\n{self.normal_organized_text}\n\ntrOCR:\n{self.trOCR_texts}")
+                return f"Google_OCR_Standard:\n{self.normal_organized_text}\n\ntrOCR:\n{self.trOCR_texts}"
+            if self.OCR_option in ['hand',]:
+                self.OCR_JSON_to_file['OCR_handwritten'] = self.hand_organized_text
+                self.OCR_JSON_to_file['OCR_trOCR'] = self.trOCR_texts
+                logger.info(f"Google_OCR_Handwriting:\n{self.hand_organized_text}\n\ntrOCR:\n{self.trOCR_texts}")
+                return f"Google_OCR_Handwriting:\n{self.hand_organized_text}\n\ntrOCR:\n{self.trOCR_texts}"
+            if self.OCR_option in ['both',]:
+                self.OCR_JSON_to_file['OCR_printed'] = self.normal_organized_text
+                self.OCR_JSON_to_file['OCR_handwritten'] = self.hand_organized_text
+                self.OCR_JSON_to_file['OCR_trOCR'] = self.trOCR_texts
+                logger.info(f"Google_OCR_Standard:\n{self.normal_organized_text}\n\nGoogle_OCR_Handwriting:\n{self.hand_organized_text}\n\ntrOCR:\n{self.trOCR_texts}")
+                return f"Google_OCR_Standard:\n{self.normal_organized_text}\n\nGoogle_OCR_Handwriting:\n{self.hand_organized_text}\n\ntrOCR:\n{self.trOCR_texts}"
+            else:
+                raise             
+
+    @staticmethod
+    def confidence_to_color(confidence):
+        hue = (confidence - 0.5) * 120 / 0.5
+        r, g, b = colorsys.hls_to_rgb(hue/360, 0.5, 1)
+        return (int(r*255), int(g*255), int(b*255))
+
+
+    def render_text_on_black_image(self, option):
+        bounds_flat = getattr(self, f'{option}_bounds_flat', [])
+        heights = getattr(self, f'{option}_height', [])
+        confidences = getattr(self, f'{option}_confidences', [])
+        characters = getattr(self, f'{option}_characters', [])
+
+        original_image = Image.open(self.path)
+        width, height = original_image.size
+        black_image = Image.new("RGB", (width, height), "black")
+        draw = ImageDraw.Draw(black_image)
+
+        for bound, confidence, char_height, character in zip(bounds_flat, confidences, heights, characters):
+            font_size = int(char_height)
+            font = ImageFont.load_default().font_variant(size=font_size)
+            if option == 'trOCR':
+                color = (0, 170, 255)
+            else:
+                color = OCRGoogle.confidence_to_color(confidence)
+            position = (bound["vertices"][0]["x"], bound["vertices"][0]["y"] - char_height)
+            draw.text(position, character, fill=color, font=font)
+
+        return black_image
+
+
+    def merge_images(self, image1, image2):
+        width1, height1 = image1.size
+        width2, height2 = image2.size
+        merged_image = Image.new("RGB", (width1 + width2, max([height1, height2])))
+        merged_image.paste(image1, (0, 0))
+        merged_image.paste(image2, (width1, 0))
+        return merged_image
+
+
+    def draw_boxes(self, option):
+        bounds = getattr(self, f'{option}_bounds', [])
+        bounds_word = getattr(self, f'{option}_bounds_word', [])
+        confidences = getattr(self, f'{option}_confidences', [])
+
+        draw = ImageDraw.Draw(self.image)
+        width, height = self.image.size
+        if min([width, height]) > 4000:
+            line_width_thick = int((width + height) / 2 * 0.0025)  # Adjust line width for character level
+            line_width_thin = 1
+        else:
+            line_width_thick = int((width + height) / 2 * 0.005)  # Adjust line width for character level
+            line_width_thin = 1 #int((width + height) / 2 * 0.001)
+
+        for bound in bounds_word:
+            draw.polygon(
+                [
+                    bound["vertices"][0]["x"], bound["vertices"][0]["y"],
+                    bound["vertices"][1]["x"], bound["vertices"][1]["y"],
+                    bound["vertices"][2]["x"], bound["vertices"][2]["y"],
+                    bound["vertices"][3]["x"], bound["vertices"][3]["y"],
+                ],
+                outline=OCRGoogle.BBOX_COLOR,
+                width=line_width_thin
+            )
+
+        # Draw a line segment at the bottom of each handwritten character
+        for bound, confidence in zip(bounds, confidences):  
+            color = OCRGoogle.confidence_to_color(confidence)
+            # Use the bottom two vertices of the bounding box for the line
+            bottom_left = (bound["vertices"][3]["x"], bound["vertices"][3]["y"] + line_width_thick)
+            bottom_right = (bound["vertices"][2]["x"], bound["vertices"][2]["y"] + line_width_thick)
+            draw.line([bottom_left, bottom_right], fill=color, width=line_width_thick)
+
+        return self.image
+
+
+    def detect_text(self):
+        client = vision.ImageAnnotatorClient()
+        with io.open(self.path, 'rb') as image_file:
+            content = image_file.read()
+        image = vision.Image(content=content)
+        response = client.document_text_detection(image=image)
+        texts = response.text_annotations
+
+        if response.error.message:
+            raise Exception(
+                '{}\nFor more info on error messages, check: '
+                'https://cloud.google.com/apis/design/errors'.format(
+                    response.error.message))
+
+        bounds = []
+        bounds_word = []
+        text_to_box_mapping = []
+        bounds_flat = []
+        height_flat = []
+        confidences = []
+        characters = []
+        organized_text = ""
+        paragraph_count = 0
+        
+        for text in texts[1:]:
+            vertices = [{"x": vertex.x, "y": vertex.y} for vertex in text.bounding_poly.vertices]
+            map_dict = {
+                "vertices": vertices,
+                "text": text.description
+            }
+            text_to_box_mapping.append(map_dict)
+
+        for page in response.full_text_annotation.pages:
+            for block in page.blocks:
+                # paragraph_count += 1
+                # organized_text += f'OCR_paragraph_{paragraph_count}:\n'  # Add paragraph label
+                for paragraph in block.paragraphs:
+
+                    avg_H_list = []
+                    for word in paragraph.words:
+                        Yw = max(vertex.y for vertex in word.bounding_box.vertices)
+                        # Calculate the width of the word and divide by the number of symbols
+                        word_length = max(vertex.x for vertex in word.bounding_box.vertices) - min(vertex.x for vertex in word.bounding_box.vertices)
+                        num_symbols = len(word.symbols)
+                        if num_symbols <= 3:
+                            H = int(Yw - min(vertex.y for vertex in word.bounding_box.vertices))
+                        else:
+                            Yo = Yw - min(vertex.y for vertex in word.bounding_box.vertices)
+                            X = word_length / num_symbols if num_symbols > 0 else 0
+                            H = int(X+(Yo*0.1))
+                        avg_H_list.append(H)
+                    avg_H = int(mean(avg_H_list))
+
+                    words_in_para = []
+                    for word in paragraph.words:
+                        # Get word-level bounding box
+                        bound_word_dict = {
+                            "vertices": [
+                                {"x": vertex.x, "y": vertex.y} for vertex in word.bounding_box.vertices
+                            ]
+                        }
+                        bounds_word.append(bound_word_dict)
+                        
+                        Y = max(vertex.y for vertex in word.bounding_box.vertices)
+                        word_x_start = min(vertex.x for vertex in word.bounding_box.vertices)
+                        word_x_end = max(vertex.x for vertex in word.bounding_box.vertices)
+                        num_symbols = len(word.symbols)
+                        symbol_width = (word_x_end - word_x_start) / num_symbols if num_symbols > 0 else 0
+
+                        current_x_position = word_x_start
+
+                        characters_ind = []
+                        for symbol in word.symbols:
+                            bound_dict = {
+                                "vertices": [
+                                    {"x": vertex.x, "y": vertex.y} for vertex in symbol.bounding_box.vertices
+                                ]
+                            }
+                            bounds.append(bound_dict)
+
+                            # Create flat bounds with adjusted x position
+                            bounds_flat_dict = {
+                                "vertices": [
+                                    {"x": current_x_position, "y": Y}, 
+                                    {"x": current_x_position + symbol_width, "y": Y}
+                                ]
+                            }
+                            bounds_flat.append(bounds_flat_dict)
+                            current_x_position += symbol_width
+
+                            height_flat.append(avg_H)
+                            confidences.append(round(symbol.confidence, 4))
+
+                            characters_ind.append(symbol.text)
+                            characters.append(symbol.text)
+
+                        words_in_para.append(''.join(characters_ind))
+                    paragraph_text = ' '.join(words_in_para)  # Join words in paragraph
+                    organized_text += paragraph_text + ' ' #+ '\n' 
+
+        # median_height = statistics.median(height_flat) if height_flat else 0
+        # median_heights = [median_height] * len(characters)
+
+        self.normal_cleaned_text = texts[0].description if texts else ''
+        self.normal_organized_text = organized_text
+        self.normal_bounds = bounds
+        self.normal_bounds_word = bounds_word
+        self.normal_text_to_box_mapping = text_to_box_mapping
+        self.normal_bounds_flat = bounds_flat
+        # self.normal_height = median_heights #height_flat
+        self.normal_height = height_flat
+        self.normal_confidences = confidences
+        self.normal_characters = characters
+
+
+    def detect_handwritten_ocr(self):
+        client = vision_beta.ImageAnnotatorClient()
+        with open(self.path, "rb") as image_file:
+            content = image_file.read()
+
+        image = vision_beta.Image(content=content)
+        image_context = vision_beta.ImageContext(language_hints=["en-t-i0-handwrit"])
+        response = client.document_text_detection(image=image, image_context=image_context)
+        texts = response.text_annotations
+        
+        if response.error.message:
+            raise Exception(
+                "{}\nFor more info on error messages, check: "
+                "https://cloud.google.com/apis/design/errors".format(response.error.message)
+            )
+
+        bounds = []
+        bounds_word = []
+        bounds_flat = []
+        height_flat = []
+        confidences = []
+        characters = []
+        organized_text = ""
+        paragraph_count = 0
+        text_to_box_mapping = []
+
+        for text in texts[1:]:
+            vertices = [{"x": vertex.x, "y": vertex.y} for vertex in text.bounding_poly.vertices]
+            map_dict = {
+                "vertices": vertices,
+                "text": text.description
+            }
+            text_to_box_mapping.append(map_dict)
+
+        for page in response.full_text_annotation.pages:
+            for block in page.blocks:
+                # paragraph_count += 1
+                # organized_text += f'\nOCR_paragraph_{paragraph_count}:\n'  # Add paragraph label
+                for paragraph in block.paragraphs:
+                    
+                    avg_H_list = []
+                    for word in paragraph.words:
+                        Yw = max(vertex.y for vertex in word.bounding_box.vertices)
+                        # Calculate the width of the word and divide by the number of symbols
+                        word_length = max(vertex.x for vertex in word.bounding_box.vertices) - min(vertex.x for vertex in word.bounding_box.vertices)
+                        num_symbols = len(word.symbols)
+                        if num_symbols <= 3:
+                            H = int(Yw - min(vertex.y for vertex in word.bounding_box.vertices))
+                        else:
+                            Yo = Yw - min(vertex.y for vertex in word.bounding_box.vertices)
+                            X = word_length / num_symbols if num_symbols > 0 else 0
+                            H = int(X+(Yo*0.1))
+                        avg_H_list.append(H)
+                    avg_H = int(mean(avg_H_list))
+
+                    words_in_para = []
+                    for word in paragraph.words:
+                        # Get word-level bounding box
+                        bound_word_dict = {
+                            "vertices": [
+                                {"x": vertex.x, "y": vertex.y} for vertex in word.bounding_box.vertices
+                            ]
+                        }
+                        bounds_word.append(bound_word_dict)
+
+                        Y = max(vertex.y for vertex in word.bounding_box.vertices)
+                        word_x_start = min(vertex.x for vertex in word.bounding_box.vertices)
+                        word_x_end = max(vertex.x for vertex in word.bounding_box.vertices)
+                        num_symbols = len(word.symbols)
+                        symbol_width = (word_x_end - word_x_start) / num_symbols if num_symbols > 0 else 0
+
+                        current_x_position = word_x_start
+
+                        characters_ind = []
+                        for symbol in word.symbols:
+                            bound_dict = {
+                                "vertices": [
+                                    {"x": vertex.x, "y": vertex.y} for vertex in symbol.bounding_box.vertices
+                                ]
+                            }
+                            bounds.append(bound_dict)
+
+                            # Create flat bounds with adjusted x position
+                            bounds_flat_dict = {
+                                "vertices": [
+                                    {"x": current_x_position, "y": Y}, 
+                                    {"x": current_x_position + symbol_width, "y": Y}
+                                ]
+                            }
+                            bounds_flat.append(bounds_flat_dict)
+                            current_x_position += symbol_width
+
+                            height_flat.append(avg_H)
+                            confidences.append(round(symbol.confidence, 4))
+
+                            characters_ind.append(symbol.text)
+                            characters.append(symbol.text)
+
+                        words_in_para.append(''.join(characters_ind))
+                    paragraph_text = ' '.join(words_in_para)  # Join words in paragraph
+                    organized_text += paragraph_text + ' ' #+ '\n' 
+
+        # median_height = statistics.median(height_flat) if height_flat else 0
+        # median_heights = [median_height] * len(characters)
+
+        self.hand_cleaned_text = response.text_annotations[0].description if response.text_annotations else ''
+        self.hand_organized_text = organized_text
+        self.hand_bounds = bounds
+        self.hand_bounds_word = bounds_word
+        self.hand_bounds_flat = bounds_flat
+        self.hand_text_to_box_mapping = text_to_box_mapping
+        # self.hand_height = median_heights #height_flat
+        self.hand_height = height_flat
+        self.hand_confidences = confidences
+        self.hand_characters = characters
+
+
+    def process_image(self, do_create_OCR_helper_image, logger):
+        if self.OCR_option in ['normal', 'both']:
+            self.detect_text()
+        if self.OCR_option in ['hand', 'both']:
+            self.detect_handwritten_ocr()
+        if self.OCR_option not in ['normal', 'hand', 'both']:
+            self.OCR_option = 'both'
+            self.detect_text()
+            self.detect_handwritten_ocr()
+
+        ### Optionally add trOCR to the self.OCR for additional context
+        self.OCR = self.detect_text_with_trOCR_using_google_bboxes(self.do_use_trOCR, logger)
+
+        if do_create_OCR_helper_image:
+            self.image = Image.open(self.path)
+
+            if self.OCR_option in ['normal', 'both']:
+                image_with_boxes_normal = self.draw_boxes('normal')
+                text_image_normal = self.render_text_on_black_image('normal')
+                self.merged_image_normal = self.merge_images(image_with_boxes_normal, text_image_normal)
+
+            if self.OCR_option in ['hand', 'both']:
+                image_with_boxes_hand = self.draw_boxes('hand')
+                text_image_hand = self.render_text_on_black_image('hand')
+                self.merged_image_hand = self.merge_images(image_with_boxes_hand, text_image_hand)
+
+            if self.do_use_trOCR:
+                text_image_trOCR = self.render_text_on_black_image('trOCR')
+
+            ### Merge final overlay image
+            ### [original, normal bboxes, normal text]
+            if self.OCR_option in ['normal']:
+                self.overlay_image = self.merge_images(Image.open(self.path), self.merged_image_normal)
+            ### [original, hand bboxes, hand text]
+            elif self.OCR_option in ['hand']:
+                self.overlay_image = self.merge_images(Image.open(self.path), self.merged_image_hand)
+            ### [original, normal bboxes, normal text, hand bboxes, hand text]
+            else:
+                self.overlay_image = self.merge_images(Image.open(self.path), self.merge_images(self.merged_image_normal, self.merged_image_hand))
+            
+            if self.do_use_trOCR:
+                self.overlay_image = self.merge_images(self.overlay_image, text_image_trOCR)
+
+        else:
+            self.merged_image_normal = None
+            self.merged_image_hand = None
+            self.overlay_image = Image.open(self.path)
+
+
+'''
+BBOX_COLOR = "black" # green cyan
+
+def render_text_on_black_image(image_path, handwritten_char_bounds_flat, handwritten_char_confidences, handwritten_char_heights, characters):
+    # Load the original image to get its dimensions
+    original_image = Image.open(image_path)
+    width, height = original_image.size
+
+    # Create a black image of the same size
+    black_image = Image.new("RGB", (width, height), "black")
+    draw = ImageDraw.Draw(black_image)
+
+    # Loop through each character
+    for bound, confidence, char_height, character in zip(handwritten_char_bounds_flat, handwritten_char_confidences, handwritten_char_heights, characters):
+        # Determine the font size based on the height of the character
+        font_size = int(char_height)
+        font = ImageFont.load_default().font_variant(size=font_size)
+
+        # Color of the character
+        color = confidence_to_color(confidence)
+
+        # Position of the text (using the bottom-left corner of the bounding box)
+        position = (bound["vertices"][0]["x"], bound["vertices"][0]["y"] - char_height)
+
+        # Draw the character
+        draw.text(position, character, fill=color, font=font)
+
+    return black_image
+
+def merge_images(image1, image2):
+    # Assuming both images are of the same size
+    width, height = image1.size
+    merged_image = Image.new("RGB", (width * 2, height))
+    merged_image.paste(image1, (0, 0))
+    merged_image.paste(image2, (width, 0))
+    return merged_image
+
 def draw_boxes(image, bounds, color):
     if bounds:
         draw = ImageDraw.Draw(image)
@@ -26,8 +590,8 @@ def draw_boxes(image, bounds, color):
             )
     return image
 
-def detect_text(path, client):
-    # client = vision.ImageAnnotatorClient()
+def detect_text(path):
+    client = vision.ImageAnnotatorClient()
     with io.open(path, 'rb') as image_file:
         content = image_file.read()
     image = vision.Image(content=content)
@@ -60,34 +624,127 @@ def detect_text(path, client):
     else:
         return '', None, None
     
-def overlay_boxes_on_image(path, bounds,do_create_OCR_helper_image):
+def confidence_to_color(confidence):
+    """Convert confidence level to a color ranging from red (low confidence) to green (high confidence)."""
+    # Using HSL color space, where Hue varies from red to green
+    hue = (confidence - 0.5) * 120 / 0.5  # Scale confidence to range 0-120 (red to green in HSL)
+    r, g, b = colorsys.hls_to_rgb(hue/360, 0.5, 1)  # Convert to RGB
+    return (int(r*255), int(g*255), int(b*255))
+
+def overlay_boxes_on_image(path, typed_bounds, handwritten_char_bounds, handwritten_char_confidences, do_create_OCR_helper_image):
     if do_create_OCR_helper_image:
         image = Image.open(path)
-        draw_boxes(image, bounds, "green")
-        return image
-    else:
-        image = Image.open(path)
-        return image
-
-
-
-
-
-
-
-
-
+        draw = ImageDraw.Draw(image)
+        width, height = image.size
+        line_width = int((width + height) / 2 * 0.005)  # Adjust line width for character level
 
+        # Draw boxes for typed text
+        for bound in typed_bounds:
+            draw.polygon(
+                [
+                    bound["vertices"][0]["x"], bound["vertices"][0]["y"],
+                    bound["vertices"][1]["x"], bound["vertices"][1]["y"],
+                    bound["vertices"][2]["x"], bound["vertices"][2]["y"],
+                    bound["vertices"][3]["x"], bound["vertices"][3]["y"],
+                ],
+                outline=BBOX_COLOR,
+                width=1
+            )
 
+        # Draw a line segment at the bottom of each handwritten character
+        for bound, confidence in zip(handwritten_char_bounds, handwritten_char_confidences):
+            color = confidence_to_color(confidence)
+            # Use the bottom two vertices of the bounding box for the line
+            bottom_left = (bound["vertices"][3]["x"], bound["vertices"][3]["y"] + line_width)
+            bottom_right = (bound["vertices"][2]["x"], bound["vertices"][2]["y"] + line_width)
+            draw.line([bottom_left, bottom_right], fill=color, width=line_width)
 
+        text_image = render_text_on_black_image(path, handwritten_char_bounds, handwritten_char_confidences)
+        merged_image = merge_images(image, text_image)  # Assuming 'overlayed_image' is the image with lines
 
 
+        return merged_image
+    else:
+        return Image.open(path)
+    
+def detect_handwritten_ocr(path):
+    """Detects handwritten characters in a local image and returns their bounding boxes and confidence levels.
 
+    Args:
+    path: The path to the local file.
 
+    Returns:
+    A tuple of (text, bounding_boxes, confidences)
+    """
+    client = vision_beta.ImageAnnotatorClient()
 
+    with open(path, "rb") as image_file:
+        content = image_file.read()
 
+    image = vision_beta.Image(content=content)
+    image_context = vision_beta.ImageContext(language_hints=["en-t-i0-handwrit"])
+    response = client.document_text_detection(image=image, image_context=image_context)
 
+    if response.error.message:
+        raise Exception(
+            "{}\nFor more info on error messages, check: "
+            "https://cloud.google.com/apis/design/errors".format(response.error.message)
+        )
 
+    bounds = []
+    bounds_flat = []
+    height_flat = []
+    confidences = []
+    character = []
+    for page in response.full_text_annotation.pages:
+        for block in page.blocks:
+            for paragraph in block.paragraphs:
+                for word in paragraph.words:
+                    # Get the bottom Y-location (max Y) for the whole word
+                    Y = max(vertex.y for vertex in word.bounding_box.vertices)
+
+                    # Get the height of the word's bounding box
+                    H = Y - min(vertex.y for vertex in word.bounding_box.vertices)
+
+                    for symbol in word.symbols:
+                        # Collecting bounding box for each symbol
+                        bound_dict = {
+                            "vertices": [
+                                {"x": vertex.x, "y": vertex.y} for vertex in symbol.bounding_box.vertices
+                            ]
+                        }
+                        bounds.append(bound_dict)
+
+                        # Bounds with same bottom y height
+                        bounds_flat_dict = {
+                            "vertices": [
+                                {"x": vertex.x, "y": Y} for vertex in symbol.bounding_box.vertices
+                            ]
+                        }
+                        bounds_flat.append(bounds_flat_dict)
+                        
+                        # Add the word's height
+                        height_flat.append(H)
+
+                        # Collecting confidence for each symbol
+                        symbol_confidence = round(symbol.confidence, 4)
+                        confidences.append(symbol_confidence)
+                        character.append(symbol.text)
+
+    cleaned_text = response.full_text_annotation.text
+
+    return cleaned_text, bounds, bounds_flat, height_flat, confidences, character
+
+
+
+def process_image(path, do_create_OCR_helper_image):
+    typed_text, typed_bounds, _ = detect_text(path)
+    handwritten_text, handwritten_bounds, _ = detect_handwritten_ocr(path)
+
+    overlayed_image = overlay_boxes_on_image(path, typed_bounds, handwritten_bounds, do_create_OCR_helper_image)
+    return typed_text, handwritten_text, overlayed_image
+
+'''
 
 # ''' Google Vision'''
 # def detect_text(path):

vouchervision/VoucherVision_Config_Builder.py

@@ -4,58 +4,103 @@ from vouchervision.general_utils import validate_dir, print_main_fail
 from vouchervision.vouchervision_main import voucher_vision
 from general_utils import get_cfg_from_full_path
 
-def build_VV_config():
-    #############################################
-    ############ Set common defaults ############
-    #############################################
-    # Changing the values below will set new 
-    # default values each time you open the 
-    # VoucherVision user interface
-    #############################################
-    #############################################
-    #############################################
+def build_VV_config(loaded_cfg=None):
+    if loaded_cfg is None:
+        #############################################
+        ############ Set common defaults ############
+        #############################################
+        # Changing the values below will set new 
+        # default values each time you open the 
+        # VoucherVision user interface
+        #############################################
+        #############################################
+        #############################################
+
+        dir_home = os.path.dirname(os.path.dirname(__file__))
+        run_name = 'test'
+        # dir_images_local = 'D:/Dropbox/LM2_Env/Image_Datasets/GBIF_BroadSample_3SppPerFamily1'
+        dir_images_local = os.path.join(dir_home,'demo','demo_images')
+        
+        # The default output location is the computer's "Downloads" folder
+        #    You can set dir_output directly by typing the folder path,
+        # OR you can uncomment the line "dir_output = default_output_folder" 
+        #    to have VoucherVision save to the Downloads folder by default
+        default_output_folder = get_default_download_folder()
+        dir_output = default_output_folder
+        # dir_output = 'D:/D_Desktop/LM2'
+
+        prefix_removal = '' #'MICH-V-'
+        suffix_removal = ''
+        catalog_numerical_only = False
+
+        save_cropped_annotations = ['label','barcode']
+
+        do_use_trOCR = False
+        OCR_option = 'hand'
+        check_for_illegal_filenames = False
+
+        LLM_version_user = 'Azure GPT 4 Turbo 1106-preview' #'Azure GPT 4 Turbo 1106-preview'
+        prompt_version = 'version_5.yaml' # from ["Version 1", "Version 1 No Domain Knowledge", "Version 2"]
+        use_LeafMachine2_collage_images = True # Use LeafMachine2 collage images
+        do_create_OCR_helper_image = True
+
+        batch_size = 500
+        num_workers = 8
+
+        path_domain_knowledge = os.path.join(dir_home,'domain_knowledge','SLTP_UM_AllAsiaMinimalInRegion.xlsx')
+        embeddings_database_name = os.path.splitext(os.path.basename(path_domain_knowledge))[0]
+
+        #############################################
+        #############################################
+        ########## DO NOT EDIT BELOW HERE ###########
+        #############################################
+        #############################################
+        return assemble_config(dir_home, run_name, dir_images_local,dir_output,
+                        prefix_removal,suffix_removal,catalog_numerical_only,LLM_version_user,batch_size,num_workers,
+                        path_domain_knowledge,embeddings_database_name,use_LeafMachine2_collage_images,
+                        prompt_version, do_create_OCR_helper_image, do_use_trOCR, OCR_option, save_cropped_annotations, 
+                        check_for_illegal_filenames, use_domain_knowledge=False)
+    else:
+        dir_home = os.path.dirname(os.path.dirname(__file__))
+        run_name = loaded_cfg['leafmachine']['project']['run_name']
+        dir_images_local = loaded_cfg['leafmachine']['project']['dir_images_local']
+        
+        default_output_folder = loaded_cfg['leafmachine']['project']['dir_output']
+        dir_output = loaded_cfg['leafmachine']['project']['dir_output']
 
-    dir_home = os.path.dirname(os.path.dirname(__file__))
-    run_name = 'test'
-    # dir_images_local = 'D:/Dropbox/LM2_Env/Image_Datasets/GBIF_BroadSample_3SppPerFamily1'
-    dir_images_local = os.path.join(dir_home,'demo','demo_images')
-    
-    # The default output location is the computer's "Downloads" folder
-    #    You can set dir_output directly by typing the folder path,
-    # OR you can uncomment the line "dir_output = default_output_folder" 
-    #    to have VoucherVision save to the Downloads folder by default
-    default_output_folder = get_default_download_folder()
-    dir_output = default_output_folder
-    # dir_output = 'D:/D_Desktop/LM2'
-
-    prefix_removal = '' #'MICH-V-'
-    suffix_removal = ''
-    catalog_numerical_only = False
+        prefix_removal = loaded_cfg['leafmachine']['project']['prefix_removal']
+        suffix_removal = loaded_cfg['leafmachine']['project']['suffix_removal']
+        catalog_numerical_only = loaded_cfg['leafmachine']['project']['catalog_numerical_only']
 
-    LLM_version_user = 'Azure GPT 4'
-    prompt_version = 'Version 2' # from ["Version 1", "Version 1 No Domain Knowledge", "Version 2"]
-    use_LeafMachine2_collage_images = False # Use LeafMachine2 collage images
-    do_create_OCR_helper_image = False
+        do_use_trOCR = loaded_cfg['leafmachine']['project']['do_use_trOCR']
+        OCR_option = loaded_cfg['leafmachine']['project']['OCR_option']
 
-    batch_size = 500
+        LLM_version_user = loaded_cfg['leafmachine']['LLM_version']
+        prompt_version = loaded_cfg['leafmachine']['project']['prompt_version']
+        use_LeafMachine2_collage_images = loaded_cfg['leafmachine']['use_RGB_label_images']
+        do_create_OCR_helper_image = loaded_cfg['leafmachine']['do_create_OCR_helper_image']
 
-    path_domain_knowledge = os.path.join(dir_home,'domain_knowledge','SLTP_UM_AllAsiaMinimalInRegion.xlsx')
-    embeddings_database_name = os.path.splitext(os.path.basename(path_domain_knowledge))[0]
+        batch_size = loaded_cfg['leafmachine']['project']['batch_size']
+        num_workers = loaded_cfg['leafmachine']['project']['num_workers']
+
+        path_domain_knowledge = loaded_cfg['leafmachine']['project']['path_to_domain_knowledge_xlsx']
+        embeddings_database_name = os.path.splitext(os.path.basename(path_domain_knowledge))[0]
+
+        save_cropped_annotations = loaded_cfg['leafmachine']['cropped_components']['save_cropped_annotations']
+        check_for_illegal_filenames = loaded_cfg['leafmachine']['do']['check_for_illegal_filenames']
+
+        return assemble_config(dir_home, run_name, dir_images_local,dir_output,
+                        prefix_removal,suffix_removal,catalog_numerical_only,LLM_version_user,batch_size,num_workers,
+                        path_domain_knowledge,embeddings_database_name,use_LeafMachine2_collage_images,
+                        prompt_version, do_create_OCR_helper_image, do_use_trOCR, OCR_option, save_cropped_annotations,
+                        check_for_illegal_filenames, use_domain_knowledge=False)
 
-    #############################################
-    #############################################
-    ########## DO NOT EDIT BELOW HERE ###########
-    #############################################
-    #############################################
-    return assemble_config(dir_home, run_name, dir_images_local,dir_output,
-                    prefix_removal,suffix_removal,catalog_numerical_only,LLM_version_user,batch_size,
-                    path_domain_knowledge,embeddings_database_name,use_LeafMachine2_collage_images,
-                    prompt_version, do_create_OCR_helper_image, use_domain_knowledge=False)
 
 def assemble_config(dir_home, run_name, dir_images_local,dir_output,
-                    prefix_removal,suffix_removal,catalog_numerical_only,LLM_version_user,batch_size,
+                    prefix_removal,suffix_removal,catalog_numerical_only,LLM_version_user,batch_size,num_workers,
                     path_domain_knowledge,embeddings_database_name,use_LeafMachine2_collage_images,
-                    prompt_version, do_create_OCR_helper_image_user, use_domain_knowledge=False):
+                    prompt_version, do_create_OCR_helper_image_user, do_use_trOCR, OCR_option, save_cropped_annotations, 
+                    check_for_illegal_filenames, use_domain_knowledge=False):
     
 
     # Initialize the base structure
@@ -65,7 +110,7 @@ def assemble_config(dir_home, run_name, dir_images_local,dir_output,
 
     # Modular sections to be added to 'leafmachine'
     do_section = {
-        'check_for_illegal_filenames': False,
+        'check_for_illegal_filenames': check_for_illegal_filenames,
         'check_for_corrupt_images_make_vertical': True,
     }
 
@@ -84,7 +129,7 @@ def assemble_config(dir_home, run_name, dir_images_local,dir_output,
         'run_name': run_name,
         'image_location': 'local',
         'batch_size': batch_size,
-        'num_workers': 1,
+        'num_workers': num_workers,
         'dir_images_local': dir_images_local,
         'continue_run_from_partial_xlsx': '',
         'prefix_removal': prefix_removal,
@@ -97,6 +142,8 @@ def assemble_config(dir_home, run_name, dir_images_local,dir_output,
         'prompt_version': prompt_version,
         'delete_all_temps': False,
         'delete_temps_keep_VVE': False,
+        'do_use_trOCR': do_use_trOCR,
+        'OCR_option': OCR_option,
     }
 
     modules_section = {
@@ -109,7 +156,7 @@ def assemble_config(dir_home, run_name, dir_images_local,dir_output,
     
     cropped_components_section = {
         'do_save_cropped_annotations': True,
-        'save_cropped_annotations': ['label','barcode'],
+        'save_cropped_annotations': save_cropped_annotations,
         'save_per_image': False,
         'save_per_annotation_class': True,
         'binarize_labels': False,
@@ -238,7 +285,7 @@ def build_api_tests(api):
                 config_data, dir_home = assemble_config(dir_home, run_name, dir_images_local,dir_output,
                     prefix_removal,suffix_removal,catalog_numerical_only,LLM_version_user,batch_size,
                     path_domain_knowledge,embeddings_database_name,use_LeafMachine2_collage_images,
-                    prompt_version, do_create_OCR_helper_image)
+                    prompt_version,do_create_OCR_helper_image)
                 
                 write_config_file(config_data, os.path.join(dir_home,'demo','demo_configs'),filename=filename)
 
@@ -264,7 +311,6 @@ def build_demo_tests(llm_version):
     batch_size = 500
     do_create_OCR_helper_image = False
 
-
     # ### Option 1: "GPT 4" of ["GPT 4", "GPT 3.5", "Azure GPT 4", "Azure GPT 3.5", "PaLM 2"]
     # LLM_version_user = 'Azure GPT 4'
     
@@ -340,7 +386,7 @@ def build_demo_tests(llm_version):
     return dir_home, path_to_configs, test_results
 
 class TestOptionsGPT:
-    OPT1 = ["GPT 4", "GPT 3.5", "Azure GPT 4", "Azure GPT 3.5"]
+    OPT1 = ["gpt-4-1106-preview","GPT 4", "GPT 3.5", "Azure GPT 4", "Azure GPT 3.5"]
     OPT2 = [False, True]
     OPT3 = ["Version 1", "Version 1 No Domain Knowledge", "Version 2"]
 
@@ -473,7 +519,7 @@ def run_demo_tests_Palm(progress_report):
         
         if check_API_key(dir_home, api_version) and check_API_key(dir_home, 'google-vision-ocr') :
             try:
-                last_JSON_response, total_cost = voucher_vision(cfg_file_path, dir_home, cfg_test=None, progress_report=progress_report, test_ind=int(test_ind))
+                last_JSON_response, total_cost = voucher_vision(cfg_file_path, dir_home, cfg_test=None, path_custom_prompts=None, progress_report=progress_report, test_ind=int(test_ind))
                 test_results[cfg] = True
                 JSON_results[ind] = last_JSON_response
             except Exception as e:
@@ -518,7 +564,7 @@ def run_api_tests(api):
             
             if check_API_key(dir_home, api) and check_API_key(dir_home, 'google-vision-ocr') :
                 try:
-                    last_JSON_response, total_cost = voucher_vision(cfg_file_path, dir_home, None, cfg_test=None, progress_report=None, test_ind=int(test_ind))
+                    last_JSON_response, total_cost = voucher_vision(cfg_file_path, dir_home, None,path_custom_prompts=None , cfg_test=None, progress_report=None, test_ind=int(test_ind))
                     test_results[cfg] = True
                     JSON_results[ind] = last_JSON_response
                     return True
@@ -532,76 +578,50 @@ def run_api_tests(api):
         print(e)
         return False
 
-# def has_API_key(val):
-#         if val != '':
-#             return True
-#         else:
-#             return False
-def has_API_key(key_name):
-    # Check if the environment variable by key_name is not None
-    return os.getenv(key_name) is not None
-
+def has_API_key(val):
+        if val != '':
+            return True
+        else:
+            return False
         
-# def check_if_usable():
-#     dir_home = os.path.dirname(os.path.dirname(__file__))
-#     path_cfg_private = os.path.join(dir_home, 'PRIVATE_DATA.yaml')
-#     cfg_private = get_cfg_from_full_path(path_cfg_private)
+def check_if_usable():
+    dir_home = os.path.dirname(os.path.dirname(__file__))
+    path_cfg_private = os.path.join(dir_home, 'PRIVATE_DATA.yaml')
+    cfg_private = get_cfg_from_full_path(path_cfg_private)
 
-#     has_key_openai = has_API_key(cfg_private['openai']['OPENAI_API_KEY'])
+    has_key_openai = has_API_key(cfg_private['openai']['OPENAI_API_KEY'])
 
-#     has_key_azure_openai = has_API_key(cfg_private['openai_azure']['api_version']) 
+    has_key_azure_openai = has_API_key(cfg_private['openai_azure']['api_version']) 
 
-#     has_key_palm2 = has_API_key(cfg_private['google_palm']['google_palm_api'])
+    has_key_palm2 = has_API_key(cfg_private['google_palm']['google_palm_api'])
     
-#     has_key_google_OCR = has_API_key(cfg_private['google_cloud']['path_json_file'])
+    has_key_google_OCR = has_API_key(cfg_private['google_cloud']['path_json_file'])
+
+    if has_key_google_OCR and (has_key_azure_openai or has_key_openai or has_key_palm2):
+        return True
+    else:
+        return False
+
+def check_API_key(dir_home, api_version):
+    dir_home = os.path.dirname(os.path.dirname(__file__))
+    path_cfg_private = os.path.join(dir_home, 'PRIVATE_DATA.yaml')
+    cfg_private = get_cfg_from_full_path(path_cfg_private)
+
+    has_key_openai = has_API_key(cfg_private['openai']['OPENAI_API_KEY'])
+
+    has_key_azure_openai = has_API_key(cfg_private['openai_azure']['api_version']) 
+
+    has_key_palm2 = has_API_key(cfg_private['google_palm']['google_palm_api'])
+
+    has_key_google_OCR = has_API_key(cfg_private['google_cloud']['path_json_file'])
 
-#     if has_key_google_OCR and (has_key_azure_openai or has_key_openai or has_key_palm2):
-#         return True
-#     else:
-#         return False
-def check_if_usable():
-    has_key_openai = os.getenv('OPENAI_API_KEY') is not None
-    has_key_palm2 = os.getenv('PALM_API_KEY') is not None
-    has_key_google_OCR = os.getenv('GOOGLE_APPLICATION_CREDENTIALS') is not None
-
-    return has_key_google_OCR and (has_key_openai or has_key_palm2)
-
-# def check_API_key(dir_home, api_version):
-#     dir_home = os.path.dirname(os.path.dirname(__file__))
-#     path_cfg_private = os.path.join(dir_home, 'PRIVATE_DATA.yaml')
-#     cfg_private = get_cfg_from_full_path(path_cfg_private)
-
-#     has_key_openai = has_API_key(cfg_private['openai']['OPENAI_API_KEY'])
-
-#     has_key_azure_openai = has_API_key(cfg_private['openai_azure']['api_version']) 
-
-#     has_key_palm2 = has_API_key(cfg_private['google_palm']['google_palm_api'])
-
-#     has_key_google_OCR = has_API_key(cfg_private['google_cloud']['path_json_file'])
-
-#     if api_version == 'palm' and has_key_palm2:
-#         return True
-#     elif api_version in ['gpt','openai'] and has_key_openai:
-#         return True
-#     elif api_version in ['gpt-azure', 'azure_openai'] and has_key_azure_openai:
-#         return True
-#     elif api_version == 'google-vision-ocr' and has_key_google_OCR:
-#         return True
-#     else:
-#         return False
-def check_API_key(api_version):
-    # The API keys are assumed to be set in the environment variables
-    has_key_openai = os.getenv('OPENAI_API_KEY') is not None
-    has_key_palm2 = os.getenv('PALM') is not None
-    has_key_google_OCR = os.getenv('GOOGLE_APPLICATION_CREDENTIALS') is not None
-
-    # Depending on the api_version, check if the corresponding key is present
     if api_version == 'palm' and has_key_palm2:
         return True
-    elif api_version in ['gpt', 'openai'] and has_key_openai:
+    elif api_version in ['gpt','openai'] and has_key_openai:
+        return True
+    elif api_version in ['gpt-azure', 'azure_openai'] and has_key_azure_openai:
         return True
     elif api_version == 'google-vision-ocr' and has_key_google_OCR:
         return True
     else:
         return False
-

vouchervision/embed_occ.py

@@ -15,7 +15,7 @@ parentdir = os.path.dirname(currentdir)
 sys.path.append(parentdir)
 from vouchervision.general_utils import get_cfg_from_full_path
 from prompts import PROMPT_UMICH_skeleton_all_asia
-from LLM_chatGPT_3_5 import num_tokens_from_string, OCR_to_dict
+from vouchervision.LLM_OpenAI import num_tokens_from_string, OCR_to_dict
 
 '''
 This generates OpenAI embedding. These are no longer used by VoucherVision.

vouchervision/embeddings_db.py

@@ -7,7 +7,7 @@ import chromadb
 from chromadb.config import Settings
 from chromadb.utils import embedding_functions
 from InstructorEmbedding import INSTRUCTOR
-from langchain.vectorstores import Chroma
+from langchain_community.vectorstores import Chroma
 ''' 
 If there is a transformers install error:
 pip install transformers==4.29.2

vouchervision/general_utils.py

@@ -9,6 +9,8 @@ import concurrent.futures
 from time import perf_counter
 import torch
 
+from vouchervision.model_maps import ModelMaps
+
 '''
 TIFF --> DNG
 Install
@@ -21,10 +23,6 @@ https://helpx.adobe.com/content/dam/help/en/photoshop/pdf/dng_commandline.pdf
 
 # https://stackoverflow.com/questions/287871/how-do-i-print-colored-text-to-the-terminal
 
-def make_zipfile(source_dir, output_filename):
-    shutil.make_archive(output_filename, 'zip', source_dir)
-    return output_filename + '.zip'
-
 def validate_dir(dir):
     if not os.path.exists(dir):
         os.makedirs(dir, exist_ok=True)
@@ -71,42 +69,42 @@ def add_to_expense_report(dir_home, data):
         # Write the data row
         writer.writerow(data)
 
-def save_token_info_as_csv(Dirs, LLM_version0, path_api_cost, total_tokens_in, total_tokens_out, n_images):
-    version_mapping = {
-            'GPT 4': 'GPT_4',
-            'GPT 3.5': 'GPT_3_5',
-            'Azure GPT 3.5': 'GPT_3_5',
-            'Azure GPT 4': 'GPT_4',
-            'PaLM 2': 'PALM2'
-        }
-    LLM_version = version_mapping[LLM_version0]
-    # Define the CSV file path
-    csv_file_path = os.path.join(Dirs.path_cost, Dirs.run_name + '.csv')
+def save_token_info_as_csv(Dirs, LLM_version0, path_api_cost, total_tokens_in, total_tokens_out, n_images, dir_home, logger):
+    if path_api_cost:
+        LLM_version = ModelMaps.get_version_mapping_cost(LLM_version0)
 
-    cost_in, cost_out, total_cost, rate_in, rate_out = calculate_cost(LLM_version, path_api_cost, total_tokens_in, total_tokens_out)
-    
-    # The data to be written to the CSV file
-    data = [Dirs.run_name, get_datetime(),LLM_version, total_cost, n_images, total_tokens_in, total_tokens_out, rate_in, rate_out, cost_in, cost_out,]
-    
-    # Open the file in write mode
-    with open(csv_file_path, mode='w', newline='') as file:
-        writer = csv.writer(file)
+        # Define the CSV file path
+        csv_file_path = os.path.join(Dirs.path_cost, Dirs.run_name + '.csv')
+
+        cost_in, cost_out, total_cost, rate_in, rate_out = calculate_cost(LLM_version, path_api_cost, total_tokens_in, total_tokens_out)
         
-        # Write the header
-        writer.writerow(['run','date','api_version','total_cost', 'n_images', 'tokens_in', 'tokens_out', 'rate_in', 'rate_out', 'cost_in', 'cost_out',])
+        # The data to be written to the CSV file
+        data = [Dirs.run_name, get_datetime(),LLM_version, total_cost, n_images, total_tokens_in, total_tokens_out, rate_in, rate_out, cost_in, cost_out,]
         
-        # Write the data
-        writer.writerow(data)
-    # Create a summary string
-    cost_summary = (f"Cost Summary for {Dirs.run_name}:\n"
-                    f"     API Cost In: ${rate_in} per 1000 Tokens\n" 
-                    f"     API Cost Out: ${rate_out} per 1000 Tokens\n" 
-                    f"     Tokens In: {total_tokens_in} - Cost: ${cost_in:.4f}\n"
-                    f"     Tokens In: {total_tokens_in} - Cost: ${cost_in:.4f}\n"
-                    f"     Tokens Out: {total_tokens_out} - Cost: ${cost_out:.4f}\n"
-                    f"     Images Processed: {n_images}\n"
-                    f"     Total Cost: ${total_cost:.4f}")
-    return cost_summary, data, total_cost
+        # Open the file in write mode
+        with open(csv_file_path, mode='w', newline='') as file:
+            writer = csv.writer(file)
+            
+            # Write the header
+            writer.writerow(['run','date','api_version','total_cost', 'n_images', 'tokens_in', 'tokens_out', 'rate_in', 'rate_out', 'cost_in', 'cost_out',])
+            
+            # Write the data
+            writer.writerow(data)
+        # Create a summary string
+        cost_summary = (f"Cost Summary for {Dirs.run_name}:\n"
+                        f"     API Cost In: ${rate_in} per 1000 Tokens\n" 
+                        f"     API Cost Out: ${rate_out} per 1000 Tokens\n" 
+                        f"     Tokens In: {total_tokens_in} - Cost: ${cost_in:.4f}\n"
+                        f"     Tokens Out: {total_tokens_out} - Cost: ${cost_out:.4f}\n"
+                        f"     Images Processed: {n_images}\n"
+                        f"     Total Cost: ${total_cost:.4f}")
+        
+        add_to_expense_report(dir_home, data)
+        logger.info(cost_summary)
+        return total_cost
+
+    else:
+        return None           #TODO add config tests to expense_report
 
 def summarize_expense_report(path_expense_report):
     # Initialize counters and sums
@@ -275,7 +273,7 @@ def create_google_ocr_yaml_config(output_file, dir_images_local, dir_output):
     }
     # Generate the YAML string from the data structure
     validate_dir(os.path.dirname(output_file))
-    yaml_str = yaml.dump(config)
+    yaml_str = yaml.dump(config, sort_keys=False)
 
     # Write the YAML string to a file
     with open(output_file, 'w') as file:
@@ -429,7 +427,7 @@ def save_config_file(cfg, logger, Dirs):
 
 def write_yaml(cfg, path_cfg):
     with open(path_cfg, 'w') as file:
-        yaml.dump(cfg, file)
+        yaml.dump(cfg, file, sort_keys=False)
 
 def split_into_batches(Project, logger, cfg):
     logger.name = 'Creating Batches'

vouchervision/model_maps.py

@@ -0,0 +1,232 @@
+class ModelMaps:
+    PROMPTS_THAT_NEED_DOMAIN_KNOWLEDGE = ["Version 1", "Version 1 PaLM 2"]
+    COLORS_EXPENSE_REPORT = {
+        'GPT_4': '#32CD32',  # Lime Green
+        'GPT_3_5': '#008000',  # Green
+        'GPT_3_5_INSTRUCT': '#3CB371',  # Medium Sea Green
+        'GPT_4_TURBO': '#228B22',  # Forest Green
+        'GPT_4_32K': '#006400',  # Dark Green
+
+        'PALM2_TB_1': '#87CEEB',  # Sky Blue
+        'PALM2_TB_2': '#1E90FF',  # Dodger Blue
+        'PALM2_TU_1': '#0000FF',  # Blue
+        'GEMINI_PRO': '#1E00FF',  # 
+
+        'AZURE_GPT_4': '#800080',  # Purple
+        'AZURE_GPT_4_TURBO': '#9370DB',  # Medium Purple
+        'AZURE_GPT_4_32K': '#8A2BE2',  # Blue Violet
+        'AZURE_GPT_3_5_INSTRUCT': '#9400D3',  # Dark Violet
+        'AZURE_GPT_3_5': '#9932CC',  # Dark Orchid
+
+        'MISTRAL_TINY': '#FFA07A',  # Light Salmon
+        'MISTRAL_SMALL': '#FF8C00',  # Dark Orange
+        'MISTRAL_MEDIUM': '#FF4500',  # Orange Red
+
+        'LOCAL_MIXTRAL_8X7B_INSTRUCT_V01': '#000000',  # Black
+        'LOCAL_MISTRAL_7B_INSTRUCT_V02': '#4a4a4a',  # Gray
+
+        'LOCAL_CPU_MISTRAL_7B_INSTRUCT_V02_GGUF': '#bababa',  # Gray
+    }
+
+    MODELS_OPENAI = ["GPT 4",
+                     "GPT 4 32k",
+                     "GPT 4 Turbo 1106-preview",
+                     "GPT 3.5",
+                     "GPT 3.5 Instruct",
+
+                     "Azure GPT 4",
+                     "Azure GPT 4 32k",
+                     "Azure GPT 4 Turbo 1106-preview",
+                     "Azure GPT 3.5",
+                     "Azure GPT 3.5 Instruct",]
+    
+    MODELS_GOOGLE = ["PaLM 2 text-bison@001",
+                     "PaLM 2 text-bison@002",
+                     "PaLM 2 text-unicorn@001",
+                     "Gemini Pro"]
+    
+    MODELS_MISTRAL = ["Mistral Tiny",
+                      "Mistral Small",
+                      "Mistral Medium",]
+
+    MODELS_LOCAL = ["LOCAL Mixtral 8x7B Instruct v0.1",
+                    "LOCAL Mistral 7B Instruct v0.2",
+                    "LOCAL CPU Mistral 7B Instruct v0.2 GGUF",]
+
+    MODELS_GUI_DEFAULT = "Azure GPT 3.5 Instruct" # "GPT 4 Turbo 1106-preview"
+
+    version_mapping_cost = {
+        'GPT 4 32k': 'GPT_4_32K',
+        'GPT 4': 'GPT_4',
+        'GPT 4 Turbo 1106-preview': 'GPT_4_TURBO',
+        'GPT 3.5 Instruct': 'GPT_3_5_INSTRUCT',
+        'GPT 3.5': 'GPT_3_5',
+
+        'Azure GPT 4 32k': 'AZURE_GPT_4_32K',
+        'Azure GPT 4': 'AZURE_GPT_4',
+        'Azure GPT 4 Turbo 1106-preview': 'AZURE_GPT_4_TURBO',
+        'Azure GPT 3.5 Instruct': 'AZURE_GPT_3_5_INSTRUCT',
+        'Azure GPT 3.5': 'AZURE_GPT_3_5',
+
+        'Gemini Pro': 'GEMINI_PRO',
+        'PaLM 2 text-unicorn@001': 'PALM2_TU_1',
+        'PaLM 2 text-bison@001': 'PALM2_TB_1',
+        'PaLM 2 text-bison@002': 'PALM2_TB_2',
+
+        'Mistral Medium': 'MISTRAL_MEDIUM',
+        'Mistral Small': 'MISTRAL_SMALL',
+        'Mistral Tiny': 'MISTRAL_TINY',
+
+        'LOCAL Mixtral 8x7B Instruct v0.1': 'LOCAL_MIXTRAL_8X7B_INSTRUCT_V01',
+        'LOCAL Mistral 7B Instruct v0.2': 'LOCAL_MISTRAL_7B_INSTRUCT_V02',
+
+        'LOCAL CPU Mistral 7B Instruct v0.2 GGUF': 'LOCAL_CPU_MISTRAL_7B_INSTRUCT_V02_GGUF',
+    }
+
+    @classmethod
+    def get_version_has_key(cls, key, has_key_openai, has_key_azure_openai, has_key_palm2, has_key_mistral):
+        # Define the mapping for 'has_key' values
+        version_has_key = {
+            'GPT 4 Turbo 1106-preview': has_key_openai,
+            'GPT 4':  has_key_openai,
+            'GPT 4 32k':  has_key_openai,
+            'GPT 3.5':  has_key_openai,
+            'GPT 3.5 Instruct':  has_key_openai,
+
+            'Azure GPT 3.5': has_key_azure_openai,
+            'Azure GPT 3.5 Instruct': has_key_azure_openai,
+            'Azure GPT 4': has_key_azure_openai,
+            'Azure GPT 4 Turbo 1106-preview': has_key_azure_openai,
+            'Azure GPT 4 32k': has_key_azure_openai,
+
+            'PaLM 2 text-bison@001':  has_key_palm2,
+            'PaLM 2 text-bison@002':  has_key_palm2,
+            'PaLM 2 text-unicorn@001':  has_key_palm2,
+            'Gemini Pro':  has_key_palm2,
+
+            'Mistral Tiny':  has_key_mistral,
+            'Mistral Small':  has_key_mistral,
+            'Mistral Medium':  has_key_mistral,
+
+            'LOCAL Mixtral 8x7B Instruct v0.1':  True,
+            'LOCAL Mistral 7B Instruct v0.2':  True,
+
+            'LOCAL CPU Mistral 7B Instruct v0.2 GGUF':  True,
+        }
+        return version_has_key.get(key)
+
+    @classmethod
+    def get_version_mapping_is_azure(cls, key):
+        version_mapping_is_azure = {
+            "GPT 4 Turbo 1106-preview": False,
+            'GPT 4': False,
+            'GPT 4 32k':  False,
+            'GPT 3.5':  False,
+            'GPT 3.5 Instruct':  False,
+
+            'Azure GPT 3.5': True,
+            'Azure GPT 3.5 Instruct': True,
+            'Azure GPT 4': True,
+            'Azure GPT 4 Turbo 1106-preview': True,
+            'Azure GPT 4 32k': True,
+
+            'PaLM 2 text-bison@001':  False,
+            'PaLM 2 text-bison@002':  False,
+            'PaLM 2 text-unicorn@001':  False,
+            'Gemini Pro':  False,
+
+            'Mistral Tiny':  False,
+            'Mistral Small':  False,
+            'Mistral Medium':  False,
+
+            'LOCAL Mixtral 8x7B Instruct v0.1':  False,
+            'LOCAL Mistral 7B Instruct v0.2':  False,
+
+            'LOCAL CPU Mistral 7B Instruct v0.2 GGUF':  False,
+        }
+        return version_mapping_is_azure.get(key)
+
+    @classmethod
+    def get_API_name(cls, key):
+        
+        ### OpenAI
+        if key == 'GPT_3_5':
+            return 'gpt-3.5-turbo-1106'
+        
+        elif key == 'GPT_3_5_INSTRUCT':
+            return 'gpt-3.5-turbo-instruct'
+        
+        elif key == 'GPT_4':
+            return 'gpt-4'
+
+        elif key == 'GPT_4_32K':
+            return 'gpt-4-32k'
+        
+        elif key == 'GPT_4_TURBO':
+            return 'gpt-4-1106-preview'
+        
+        ### Azure
+        elif key == 'AZURE_GPT_3_5':
+            return 'gpt-35-turbo-1106'
+
+        elif key == 'AZURE_GPT_3_5_INSTRUCT':
+            return 'gpt-35-turbo-instruct'
+        
+        elif key == 'AZURE_GPT_4':
+            return "gpt-4"
+    
+        elif key == 'AZURE_GPT_4_TURBO':
+            return "gpt-4-1106-preview"
+        
+        elif key == 'AZURE_GPT_4_32K':
+            return "gpt-4-32k"
+        
+        ### Google
+        elif key == 'PALM2_TB_1':
+            return "text-bison@001"
+        
+        elif key == 'PALM2_TB_2':
+            return "text-bison@002"
+        
+        elif key == 'PALM2_TU_1':
+            return "text-unicorn@001"
+        
+        elif key == 'GEMINI_PRO':
+            return "gemini-pro"
+        
+        ### Mistral 
+        elif key == 'MISTRAL_TINY':
+            return "mistral-tiny"
+        
+        elif key == 'MISTRAL_SMALL':
+            return 'mistral-small'
+        
+        elif key == 'MISTRAL_MEDIUM':
+            return 'mistral-medium'
+        
+
+        ### Mistral LOCAL
+        elif key == 'LOCAL_MIXTRAL_8X7B_INSTRUCT_V01':
+            return 'Mixtral-8x7B-Instruct-v0.1'
+        
+        elif key == 'LOCAL_MISTRAL_7B_INSTRUCT_V02':
+            return 'Mistral-7B-Instruct-v0.2'
+        
+        ### Mistral LOCAL CPU
+        elif key == 'LOCAL_CPU_MISTRAL_7B_INSTRUCT_V02_GGUF':
+            return 'Mistral-7B-Instruct-v0.2-GGUF'
+
+        else:
+            raise ValueError(f"Invalid model name {key}. See model_maps.py") 
+
+    @classmethod
+    def get_models_gui_list(cls):
+        return cls.MODELS_LOCAL + cls.MODELS_GOOGLE + cls.MODELS_OPENAI + cls.MODELS_MISTRAL
+
+    @classmethod
+    def get_version_mapping_cost(cls, key):
+        return cls.version_mapping_cost.get(key, None)
+    
+    @classmethod
+    def get_all_mapping_cost(cls):
+        return cls.version_mapping_cost

vouchervision/prompt_catalog.py

@@ -1,9 +1,7 @@
 from dataclasses import dataclass
+from langchain_core.pydantic_v1 import Field, create_model
 import yaml, json
 
-
-#  catalog = PromptCatalog(OCR="Sample OCR text", domain_knowledge_example="Sample domain knowledge", similarity="0.9")
-
 @dataclass
 class PromptCatalog:
     domain_knowledge_example: str = ""
@@ -11,565 +9,6 @@ class PromptCatalog:
     OCR: str = ""
     n_fields: int = 0
 
-    # def PROMPT_UMICH_skeleton_all_asia(self, OCR=None, domain_knowledge_example=None, similarity=None):
-    def prompt_v1_verbose(self, OCR=None, domain_knowledge_example=None, similarity=None):
-        self.OCR = OCR or self.OCR
-        self.domain_knowledge_example = domain_knowledge_example or self.domain_knowledge_example
-        self.similarity = similarity or self.similarity
-        self.n_fields = 22 or self.n_fields
-
-        set_rules = """
-        Please note that your task is to generate a dictionary, following the below rules:
-        1. Refactor the unstructured OCR text into a dictionary based on the reference dictionary structure (ref_dict).
-        2. Each field of OCR corresponds to a column of the ref_dict. You should correctly map the values from OCR to the respective fields in ref_dict.
-        3. If the OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None".
-        4. If there is a field in the ref_dict that does not have a corresponding value in the OCR text, fill it based on your knowledge but don't generate new information.
-        5. Do not use any text from the ref_dict values in the new dict, but you must use the headers from ref_dict.
-        6. Duplicate dictionary fields are not allowed.
-        7. Only return the new dictionary. You should not explain your answer.
-        8. Your output should be a Python dictionary represented as a JSON string.
-        """
-
-        umich_all_asia_rules = """{
-            "Catalog Number": {
-                "format": "[Catalog Number]",
-                "null_value": "",
-                "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits"
-            },
-            "Genus": {
-                "format": "[Genus] or '[Family] indet' if no genus",
-                "null_value": "",
-                "description": "Taxonomic determination to genus, do capitalize genus"
-            },
-            "Species": {
-                "format": "[species] or 'indet' if no species",
-                "null_value": "",
-                "description": "Taxonomic determination to species, do not capitalize species"
-            },
-            "subspecies": {
-                "format": "[subspecies]",
-                "null_value": "",
-                "description": "Taxonomic determination to subspecies (subsp.)"
-            },
-            "variety": {
-                "format": "[variety]",
-                "null_value": "",
-                "description": "Taxonomic determination to variety (var)"
-            },
-            "forma": {
-                "format": "[form]",
-                "null_value": "",
-                "description": "Taxonomic determination to form (f.)"
-            },
-            "Country": {
-                "format": "[Country]",
-                "null_value": "",
-                "description": "Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abbreviation is given"
-            },
-            "State": {
-                "format": "[Adm. Division 1]",
-                "null_value": "",
-                "description": "Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word"
-            },
-            "County": {
-                "format": "[Adm. Division 2]",
-                "null_value": "",
-                "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word"
-            },
-            "Locality Name": {
-                "format": "verbatim, if no geographic info: 'no data provided on label of catalog no: [######]', or if illegible: 'locality present but illegible/not translated for catalog no: #######', or if no named locality: 'no named locality for catalog no: #######'",
-                "description": "Description of geographic location or landscape"
-            },
-            "Min Elevation": {
-                "format": "elevation integer",
-                "null_value": "",
-                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given"
-            },
-            "Max Elevation": {
-                "format": "elevation integer",
-                "null_value": "",
-                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise"
-            },
-            "Elevation Units": {
-                "format": "m",
-                "null_value": "",
-                "description": "'m' only if an elevation is present"
-            },
-            "Verbatim Coordinates": {
-                "format": "[Lat, Long | UTM | TRS]",
-                "null_value": "",
-                "description": "Verbatim coordinates as they appear on the label, fix typos to match standardized GPS coordinate format"
-            },
-            "Datum": {
-                "format": "[WGS84, NAD23 etc.]",
-                "null_value": "",
-                "description": "GPS Datum of coordinates on label; empty string "" if GPS coordinates are not in OCR"
-            },
-            "Cultivated": {
-                "format": "yes",
-                "null_value": "",
-                "description": "Indicates if specimen was grown in cultivation"
-            },
-            "Habitat": {
-                "format": "verbatim",
-                "null_value": "",
-                "description": "Description of habitat or location where specimen was collected, ignore descriptions of the plant itself"
-            },
-            "Collectors": {
-                "format": "[Collector]",
-                "null_value": "not present",
-                "description": "Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas"
-            },
-            "Collector Number": {
-                "format": "[Collector No.]",
-                "null_value": "s.n.",
-                "description": "Sequential number assigned to collection, associated with the collector"
-            },
-            "Verbatim Date": {
-                "format": "verbatim",
-                "null_value": "s.d.",
-                "description": "Date of collection exactly as it appears on the label"
-            },
-            "Date": {
-                "format": "[yyyy-mm-dd]",
-                "null_value": "",
-                "description": "Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e., 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day"
-            },
-            "End Date": {
-                "format": "[yyyy-mm-dd]",
-                "null_value": "",
-                "description": "If date range is listed, later date of collection range"
-            }
-        }"""
-
-        structure = """{"Dictionary":
-                            {
-                            "Catalog Number": [Catalog Number],
-                            "Genus": [Genus],
-                            "Species": [species],
-                            "subspecies": [subspecies],
-                            "variety": [variety],
-                            "forma": [forma],
-                            "Country": [Country],
-                            "State": [State],
-                            "County": [County],
-                            "Locality Name": [Locality Name],
-                            "Min Elevation": [Min Elevation],
-                            "Max Elevation": [Max Elevation],
-                            "Elevation Units": [Elevation Units],
-                            "Verbatim Coordinates": [Verbatim Coordinates],
-                            "Datum": [Datum],
-                            "Cultivated": [Cultivated],
-                            "Habitat": [Habitat],
-                            "Collectors": [Collectors],
-                            "Collector Number": [Collector Number],
-                            "Verbatim Date": [Verbatim Date],
-                            "Date": [Date],
-                            "End Date": [End Date]
-                            },
-                        "SpeciesName": {"taxonomy": [Genus_species]}}"""
-
-        prompt = f"""I'm providing you with a set of rules, an unstructured OCR text, and a reference dictionary (domain knowledge). Your task is to convert the OCR text into a structured dictionary that matches the structure of the reference dictionary. Please follow the rules strictly.
-        The rules are as follows:
-        {set_rules}
-        The unstructured OCR text is:
-        {self.OCR}
-        The reference dictionary, which provides an example of the output structure and has an embedding distance of {self.similarity} to the OCR, is:
-        {self.domain_knowledge_example}
-        Some dictionary fields have special requirements. These requirements specify the format for each field, and are given below:
-        {umich_all_asia_rules}
-        Please refactor the OCR text into a dictionary, following the rules and the reference structure:
-        {structure}
-        """
-
-        xlsx_headers = ["Catalog Number","Genus","Species","subspecies","variety","forma","Country","State","County","Locality Name","Min Elevation","Max Elevation","Elevation Units","Verbatim Coordinates","Datum","Cultivated","Habitat","Collectors","Collector Number","Verbatim Date","Date","End Date"]
-
-
-        return prompt, self.n_fields, xlsx_headers
-    
-    def prompt_v1_verbose_noDomainKnowledge(self, OCR=None):
-        self.OCR = OCR or self.OCR
-        self.n_fields = 22 or self.n_fields
-
-        set_rules = """
-        Please note that your task is to generate a dictionary, following the below rules:
-        1. Refactor the unstructured OCR text into a dictionary based on the reference dictionary structure (ref_dict).
-        2. Each field of OCR corresponds to a column of the ref_dict. You should correctly map the values from OCR to the respective fields in ref_dict.
-        3. If the OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None".
-        4. If there is a field in the ref_dict that does not have a corresponding value in the OCR text, fill it based on your knowledge but don't generate new information.
-        5. Do not use any text from the ref_dict values in the new dict, but you must use the headers from ref_dict.
-        6. Duplicate dictionary fields are not allowed.
-        7. Only return the new dictionary. You should not explain your answer.
-        8. Your output should be a Python dictionary represented as a JSON string.
-        """
-
-        umich_all_asia_rules = """{
-            "Catalog Number": {
-                "format": "[Catalog Number]",
-                "null_value": "",
-                "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits"
-            },
-            "Genus": {
-                "format": "[Genus] or '[Family] indet' if no genus",
-                "null_value": "",
-                "description": "Taxonomic determination to genus, do capitalize genus"
-            },
-            "Species": {
-                "format": "[species] or 'indet' if no species",
-                "null_value": "",
-                "description": "Taxonomic determination to species, do not capitalize species"
-            },
-            "subspecies": {
-                "format": "[subspecies]",
-                "null_value": "",
-                "description": "Taxonomic determination to subspecies (subsp.)"
-            },
-            "variety": {
-                "format": "[variety]",
-                "null_value": "",
-                "description": "Taxonomic determination to variety (var)"
-            },
-            "forma": {
-                "format": "[form]",
-                "null_value": "",
-                "description": "Taxonomic determination to form (f.)"
-            },
-            "Country": {
-                "format": "[Country]",
-                "null_value": "",
-                "description": "Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abbreviation is given"
-            },
-            "State": {
-                "format": "[Adm. Division 1]",
-                "null_value": "",
-                "description": "Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word"
-            },
-            "County": {
-                "format": "[Adm. Division 2]",
-                "null_value": "",
-                "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word"
-            },
-            "Locality Name": {
-                "format": "verbatim, if no geographic info: 'no data provided on label of catalog no: [######]', or if illegible: 'locality present but illegible/not translated for catalog no: #######', or if no named locality: 'no named locality for catalog no: #######'",
-                "description": "Description of geographic location or landscape"
-            },
-            "Min Elevation": {
-                "format": "elevation integer",
-                "null_value": "",
-                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given"
-            },
-            "Max Elevation": {
-                "format": "elevation integer",
-                "null_value": "",
-                "description": "Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise"
-            },
-            "Elevation Units": {
-                "format": "m",
-                "null_value": "",
-                "description": "'m' only if an elevation is present"
-            },
-            "Verbatim Coordinates": {
-                "format": "[Lat, Long | UTM | TRS]",
-                "null_value": "",
-                "description": "Verbatim coordinates as they appear on the label, fix typos to match standardized GPS coordinate format"
-            },
-            "Datum": {
-                "format": "[WGS84, NAD23 etc.]",
-                "null_value": "",
-                "description": "GPS Datum of coordinates on label; empty string "" if GPS coordinates are not in OCR"
-            },
-            "Cultivated": {
-                "format": "yes",
-                "null_value": "",
-                "description": "Indicates if specimen was grown in cultivation"
-            },
-            "Habitat": {
-                "format": "verbatim",
-                "null_value": "",
-                "description": "Description of habitat or location where specimen was collected, ignore descriptions of the plant itself"
-            },
-            "Collectors": {
-                "format": "[Collector]",
-                "null_value": "not present",
-                "description": "Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas"
-            },
-            "Collector Number": {
-                "format": "[Collector No.]",
-                "null_value": "s.n.",
-                "description": "Sequential number assigned to collection, associated with the collector"
-            },
-            "Verbatim Date": {
-                "format": "verbatim",
-                "null_value": "s.d.",
-                "description": "Date of collection exactly as it appears on the label"
-            },
-            "Date": {
-                "format": "[yyyy-mm-dd]",
-                "null_value": "",
-                "description": "Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e., 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day"
-            },
-            "End Date": {
-                "format": "[yyyy-mm-dd]",
-                "null_value": "",
-                "description": "If date range is listed, later date of collection range"
-            }
-        }"""
-
-        structure = """{"Dictionary":
-                            {
-                            "Catalog Number": [Catalog Number],
-                            "Genus": [Genus],
-                            "Species": [species],
-                            "subspecies": [subspecies],
-                            "variety": [variety],
-                            "forma": [forma],
-                            "Country": [Country],
-                            "State": [State],
-                            "County": [County],
-                            "Locality Name": [Locality Name],
-                            "Min Elevation": [Min Elevation],
-                            "Max Elevation": [Max Elevation],
-                            "Elevation Units": [Elevation Units],
-                            "Verbatim Coordinates": [Verbatim Coordinates],
-                            "Datum": [Datum],
-                            "Cultivated": [Cultivated],
-                            "Habitat": [Habitat],
-                            "Collectors": [Collectors],
-                            "Collector Number": [Collector Number],
-                            "Verbatim Date": [Verbatim Date],
-                            "Date": [Date],
-                            "End Date": [End Date]
-                            },
-                        "SpeciesName": {"taxonomy": [Genus_species]}}"""
-
-        prompt = f"""I'm providing you with a set of rules, an unstructured OCR text, and a reference dictionary (domain knowledge). Your task is to convert the OCR text into a structured dictionary that matches the structure of the reference dictionary. Please follow the rules strictly.
-        The rules are as follows:
-        {set_rules}
-        The unstructured OCR text is:
-        {self.OCR}
-        Some dictionary fields have special requirements. These requirements specify the format for each field, and are given below:
-        {umich_all_asia_rules}
-        Please refactor the OCR text into a dictionary, following the rules and the reference structure:
-        {structure}
-        """
-
-        xlsx_headers = ["Catalog Number","Genus","Species","subspecies","variety","forma","Country","State","County","Locality Name","Min Elevation","Max Elevation","Elevation Units","Verbatim Coordinates","Datum","Cultivated","Habitat","Collectors","Collector Number","Verbatim Date","Date","End Date"]
-
-        return prompt, self.n_fields, xlsx_headers
-    
-    def prompt_v2_json_rules(self, OCR=None):
-        self.OCR = OCR or self.OCR
-        self.n_fields = 26 or self.n_fields
-
-        set_rules = """
-        1. Refactor the unstructured OCR text into a dictionary based on the JSON structure outlined below.
-        2. You should map the unstructured OCR text to the appropriate JSON key and then populate the field based on its rules.
-        3. Some JSON key fields are permitted to remain empty if the corresponding information is not found in the unstructured OCR text.
-        4. Ignore any information in the OCR text that doesn't fit into the defined JSON structure.
-        5. Duplicate dictionary fields are not allowed.
-        6. Ensure that all JSON keys are in lowercase.
-        7. Ensure that new JSON field values follow sentence case capitalization.
-        7. Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-        8. Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-        9. Only return a JSON dictionary represented as a string. You should not explain your answer.
-        """
-
-        dictionary_field_format_descriptions = """
-        The next section of instructions outlines how to format the JSON dictionary. The keys are the same as those of the final formatted JSON object.
-        For each key there is a format requirement that specifies how to transcribe the information for that key. 
-        The possible formatting options are:
-        1. "verbatim transcription" - field is populated with verbatim text from the unformatted OCR.
-        2. "spell check transcription" - field is populated with spelling corrected text from the unformatted OCR.
-        3. "boolean yes no" - field is populated with only yes or no.
-        4. "integer" - field is populated with only an integer.
-        5. "[list]" - field is populated from one of the values in the list.
-        6. "yyyy-mm-dd" - field is populated with a date in the format year-month-day.
-        The desired null value is also given. Populate the field with the null value of the information for that key is not present in the unformatted OCR text. 
-        """
-
-        json_template_rules = """
-            {"Dictionary":{
-                "catalog_number": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits."
-                },
-                "genus": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to genus. Genus must be capitalized. If genus is not present use the taxonomic family name followed by the word 'indet'."
-                },
-                "species": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to species, do not capitalize species."
-                },
-                "subspecies": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to subspecies (subsp.)."
-                },
-                "variety": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to variety (var)."
-                },
-                "forma": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to form (f.)."
-                },
-                "country": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Country that corresponds to the current geographic location of collection. Capitalize first letter of each word. If abbreviation is given populate field with the full spelling of the country's name."
-                },
-                "state": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Administrative division 1 that corresponds to the current geographic location of collection. Capitalize first letter of each word. Administrative division 1 is equivalent to a U.S. State."
-                },
-                "county": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word. Administrative division 2 is equivalent to a U.S. county, parish, borough."
-                },
-                "locality_name": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Description of geographic location, landscape, landmarks, regional features, nearby places, or any contextual information aiding in pinpointing the exact origin or site of the specimen."
-                },
-                "min_elevation": {
-                    "format": "integer",
-                    "null_value": "",
-                    "description": "Minimum elevation or altitude in meters. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
-                },
-                "max_elevation": {
-                    "format": "integer",
-                    "null_value": "",
-                    "description": "Maximum elevation or altitude in meters. If only one elevation is present, then max_elevation should be set to the null_value. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
-                },
-                "elevation_units": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Elevation units must be meters. If min_elevation field is populated, then elevation_units: 'm'. Otherwise elevation_units: ''."
-                },
-                "verbatim_coordinates": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Verbatim location coordinates as they appear on the label. Do not convert formats. Possible coordinate types are one of [Lat, Long, UTM, TRS]."
-                },
-                "decimal_coordinates": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Correct and convert the verbatim location coordinates to conform with the decimal degrees GPS coordinate format."
-                },
-                "datum": {
-                    "format": "[WGS84, WGS72, WGS66, WGS60, NAD83, NAD27, OSGB36, ETRS89, ED50, GDA94, JGD2011, Tokyo97, KGD2002, TWD67, TWD97, BJS54, XAS80, GCJ-02, BD-09, PZ-90.11, GTRF, CGCS2000, ITRF88, ITRF89, ITRF90, ITRF91, ITRF92, ITRF93, ITRF94, ITRF96, ITRF97, ITRF2000, ITRF2005, ITRF2008, ITRF2014, Hong Kong Principal Datum, SAD69]",
-                    "null_value": "",
-                    "description": "Datum of location coordinates. Possible values are include in the format list. Leave field blank if unclear."
-                },
-                "cultivated": {
-                    "format": "boolean yes no",
-                    "null_value": "",
-                    "description": "Cultivated plants are intentionally grown by humans. In text descriptions, look for planting dates, garden locations, ornamental, cultivar names, garden, or farm to indicate cultivated plant."
-                },
-                "habitat": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Description of a plant's habitat or the location where the specimen was collected. Ignore descriptions of the plant itself."
-                },
-                "plant_description": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Description of plant features such as leaf shape, size, color, stem texture, height, flower structure, scent, fruit or seed characteristics, root system type, overall growth habit and form, any notable aroma or secretions, presence of hairs or bristles, and any other distinguishing morphological or physiological characteristics."
-                },
-                "collectors": {
-                    "format": "verbatim transcription",
-                    "null_value": "not present",
-                    "description": "Full name(s) of the individual(s) responsible for collecting the specimen. When multiple collectors are involved, their names should be separated by commas."
-                },
-                "collector_number": {
-                    "format": "verbatim transcription",
-                    "null_value": "s.n.",
-                    "description": "Unique identifier or number that denotes the specific collecting event and associated with the collector."
-                },
-                "determined_by": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Full name of the individual responsible for determining the taxanomic name of the specimen. Sometimes the name will be near to the characters 'det' to denote determination. This name may be isolated from other names in the unformatted OCR text."
-                },
-                "multiple_names": {
-                    "format": "boolean yes no",
-                    "null_value": "",
-                    "description": "Indicate whether multiple people or collector names are present in the unformatted OCR text. If you see more than one person's name the value is 'yes'; otherwise the value is 'no'."
-                },
-                "verbatim_date": {
-                    "format": "verbatim transcription",
-                    "null_value": "s.d.",
-                    "description": "Date of collection exactly as it appears on the label. Do not change the format or correct typos."
-                },
-                "date": {
-                    "format": "yyyy-mm-dd",
-                    "null_value": "",
-                    "description": "Date the specimen was collected formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire date is unknown, 'YYYY-00-00' if only the year is known, and 'YYYY-MM-00' if year and month are known but day is not."
-                },
-                "end_date": {
-                    "format": "yyyy-mm-dd",
-                    "null_value": "",
-                    "description": "If a date range is provided, this represents the later or ending date of the collection period, formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire end date is unknown, 'YYYY-00-00' if only the year of the end date is known, and 'YYYY-MM-00' if year and month of the end date are known but the day is not."
-                },
-            },
-            "SpeciesName": {
-                "taxonomy": [Genus_species]}
-            }"""
-        
-        structure = """{"Dictionary":
-                            {
-                            "catalog_number": "",
-                            "genus": "",
-                            "species": "",
-                            "subspecies": "",
-                            "variety": "",
-                            "forma": "",
-                            "country": "",
-                            "state": "",
-                            "county": "",
-                            "locality_name": "",
-                            "min_elevation": "",
-                            "max_elevation": "",
-                            "elevation_units": "",
-                            "verbatim_coordinates": "",
-                            "decimal_coordinates": "",
-                            "datum": "",
-                            "cultivated": "",
-                            "habitat": "",
-                            "plant_description": "",
-                            "collectors": "",
-                            "collector_number": "",
-                            "determined_by": "",
-                            "multiple_names": "",
-                            "verbatim_date":"" ,
-                            "date": "",
-                            "end_date": ""
-                            },
-                        "SpeciesName": {"taxonomy": ""}}"""
-
-        prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
-        The rules are:
-        {set_rules}
-        The unstructured OCR text is:
-        {self.OCR}
-        {dictionary_field_format_descriptions}
-        This is the JSON template that includes instructions for each key:
-        {json_template_rules}
-        Please populate the following JSON dictionary based on the rules and the unformatted OCR text:
-        {structure}
-        """
-
-        xlsx_headers = ["catalog_number","genus","species","subspecies","variety","forma","country","state","county","locality_name","min_elevation","max_elevation","elevation_units","verbatim_coordinates","decimal_coordinates","datum","cultivated","habitat","plant_description","collectors","collector_number","determined_by","multiple_names","verbatim_date","date","end_date"]
-        
-        return prompt, self.n_fields, xlsx_headers
     
     #############################################################################################
     #############################################################################################
@@ -578,7 +17,7 @@ class PromptCatalog:
     # These are for dynamically creating your own prompts with n-columns
 
 
-    def prompt_v2_custom(self, rules_config_path, OCR=None, is_palm=False):
+    def prompt_SLTP(self, rules_config_path, OCR=None, is_palm=False):
         self.OCR = OCR
 
         self.rules_config_path = rules_config_path
@@ -588,22 +27,26 @@ class PromptCatalog:
         self.json_formatting_instructions = self.rules_config['json_formatting_instructions']
 
         self.rules_list = self.rules_config['rules']
-        self.n_fields = len(self.rules_list['Dictionary'])
+        self.n_fields = len(self.rules_config['rules'])
 
         # Set the rules for processing OCR into JSON format
         self.rules = self.create_rules(is_palm)
 
-        self.structure = self.create_structure(is_palm)
+        self.structure, self.dictionary_structure = self.create_structure(is_palm)
 
+        ''' between  instructions and json_formatting_instructions. Made the prompt too long. Better performance without it
+        The unstructured OCR text is:
+        {self.OCR}
+        '''
         if is_palm:
             prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
                 The rules are:
                 {self.instructions}
-                The unstructured OCR text is:
-                {self.OCR}
                 {self.json_formatting_instructions}
                 This is the JSON template that includes instructions for each key:
                 {self.rules}
+                The unstructured OCR text is:
+                {self.OCR}
                 Please populate the following JSON dictionary based on the rules and the unformatted OCR text:
                 {self.structure}
                 {self.structure}
@@ -613,17 +56,18 @@ class PromptCatalog:
             prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
                 The rules are:
                 {self.instructions}
-                The unstructured OCR text is:
-                {self.OCR}
                 {self.json_formatting_instructions}
                 This is the JSON template that includes instructions for each key:
                 {self.rules}
+                The unstructured OCR text is:
+                {self.OCR}
                 Please populate the following JSON dictionary based on the rules and the unformatted OCR text:
                 {self.structure}
                 """
-        xlsx_headers = self.generate_xlsx_headers(is_palm)
+        # xlsx_headers = self.generate_xlsx_headers(is_palm)
         
-        return prompt, self.n_fields, xlsx_headers
+        # return prompt, self.PromptJSONModel, self.n_fields, xlsx_headers
+        return prompt, self.dictionary_structure
 
     def load_rules_config(self):
         with open(self.rules_config_path, 'r') as stream:
@@ -634,64 +78,31 @@ class PromptCatalog:
                 return None
 
     def create_rules(self, is_palm=False):
-        if is_palm:
-            # Start with a structure for the "Dictionary" section where each key contains its rules
-            dictionary_structure = {
-                key: {
-                    'description': value['description'],
-                    'format': value['format'],
-                    'null_value': value.get('null_value', '')
-                } for key, value in self.rules_list['Dictionary'].items()
-            }
-
-            # Convert the structure to a JSON string without indentation
-            structure_json_str = json.dumps(dictionary_structure, sort_keys=False)
-            return structure_json_str
-        
-        else:
-            # Start with a structure for the "Dictionary" section where each key contains its rules
-            dictionary_structure = {
-                key: {
-                    'description': value['description'],
-                    'format': value['format'],
-                    'null_value': value.get('null_value', '')
-                } for key, value in self.rules_list['Dictionary'].items()
-            }
-
-            # Combine both sections into the overall structure
-            full_structure = {
-                "Dictionary": dictionary_structure,
-                "SpeciesName": self.rules_list['SpeciesName']
-            }
+        dictionary_structure = {key: value for key, value in self.rules_list.items()}
 
-            # Convert the structure to a JSON string without indentation
-            structure_json_str = json.dumps(full_structure, sort_keys=False)
-            return structure_json_str
+        # Convert the structure to a JSON string without indentation
+        structure_json_str = json.dumps(dictionary_structure, sort_keys=False)
+        return structure_json_str
     
     def create_structure(self, is_palm=False):
-        if is_palm:
-            # Start with an empty structure for the "Dictionary" section
-            dictionary_structure = {key: "" for key in self.rules_list['Dictionary'].keys()}
+        # Create fields for the Pydantic model dynamically
+        fields = {key: (str, Field(default=value, description=value)) for key, value in self.rules_list.items()}
 
-            # Convert the structure to a JSON string with indentation for readability
-            structure_json_str = json.dumps(dictionary_structure, sort_keys=False)
-            return structure_json_str
-        else:
-            # Start with an empty structure for the "Dictionary" section
-            dictionary_structure = {key: "" for key in self.rules_list['Dictionary'].keys()}
-            
-            # Manually define the "SpeciesName" section
-            species_name_structure = {"taxonomy": ""}
+        # Dynamically create the Pydantic model
+        DynamicJSONParsingModel = create_model('SLTPvA', **fields)
+        DynamicJSONParsingModel_use = DynamicJSONParsingModel()
+
+        # Define the structure for the "Dictionary" section
+        dictionary_fields = {key: (str, Field(default='', description="")) for key in self.rules_list.keys()}
+        
+        # Dynamically create the "Dictionary" Pydantic model
+        PromptJSONModel = create_model('PromptJSONModel', **dictionary_fields)
 
-            # Combine both sections into the overall structure
-            full_structure = {
-                "Dictionary": dictionary_structure,
-                "SpeciesName": species_name_structure
-            }
+        # Convert the model to JSON string (for demonstration)
+        dictionary_structure = PromptJSONModel().dict()
+        structure_json_str = json.dumps(dictionary_structure, sort_keys=False, indent=4)
+        return structure_json_str, dictionary_structure
 
-            # Convert the structure to a JSON string with indentation for readability
-            structure_json_str = json.dumps(full_structure, sort_keys=False)
-            return structure_json_str
 
     def generate_xlsx_headers(self, is_palm):
         # Extract headers from the 'Dictionary' keys in the JSON template rules
@@ -699,617 +110,6 @@ class PromptCatalog:
             xlsx_headers = list(self.rules_list.keys())
             return xlsx_headers     
         else:
-            xlsx_headers = list(self.rules_list["Dictionary"].keys())
+            xlsx_headers = list(self.rules_list.keys())
             return xlsx_headers
     
-    def prompt_v2_custom_redo(self, incorrect_json, is_palm):
-        # Load the existing rules and structure
-        self.rules_config = self.load_rules_config()
-        # self.rules = self.create_rules(is_palm)
-        self.structure = self.create_structure(is_palm)
-        
-        # Generate the prompt using the loaded rules and structure
-        if is_palm:
-            prompt = f"""The incorrectly formatted JSON dictionary below is not valid. It contains an error that prevents it from loading with the Python command json.loads().
-                The incorrectly formatted JSON dictionary below is the literal string returned by a previous function and the error may be caused by markdown formatting.
-                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
-                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-                The incorrectly formatted JSON dictionary: {incorrect_json}
-                The output JSON structure: {self.structure}
-                The output JSON structure: {self.structure}
-                The output JSON structure: {self.structure}
-                Please reformat the incorrectly formatted JSON dictionary given the output JSON structure: """
-        else:
-            prompt = f"""The incorrectly formatted JSON dictionary below is not valid. It contains an error that prevents it from loading with the Python command json.loads().
-                The incorrectly formatted JSON dictionary below is the literal string returned by a previous function and the error may be caused by markdown formatting.
-                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
-                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-                The incorrectly formatted JSON dictionary: {incorrect_json}
-                The output JSON structure: {self.structure}
-                Please reformat the incorrectly formatted JSON dictionary given the output JSON structure: """
-        return prompt
-
-    #############################################################################################
-    #############################################################################################
-    #############################################################################################
-    #############################################################################################
-    def prompt_gpt_redo_v1(self, incorrect_json):       
-        structure = """Below is the correct JSON formatting. Modify the text to conform to the following format, fixing the incorrect JSON:
-        {"Dictionary":
-            {
-            "Catalog Number": [Catalog Number],
-            "Genus": [Genus],
-            "Species": [species],
-            "subspecies": [subspecies],
-            "variety": [variety],
-            "forma": [forma],
-            "Country": [Country],
-            "State": [State],
-            "County": [County],
-            "Locality Name": [Locality Name],
-            "Min Elevation": [Min Elevation],
-            "Max Elevation": [Max Elevation],
-            "Elevation Units": [Elevation Units],
-            "Verbatim Coordinates": [Verbatim Coordinates],
-            "Datum": [Datum],
-            "Cultivated": [Cultivated],
-            "Habitat": [Habitat],
-            "Collectors": [Collectors],
-            "Collector Number": [Collector Number],
-            "Verbatim Date": [Verbatim Date],
-            "Date": [Date],
-            "End Date": [End Date]
-            },
-        "SpeciesName": {"taxonomy": [Genus_species]}}"""
-
-        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
-                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
-                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-                The incorrectly formatted JSON dictionary: {incorrect_json}
-                The output JSON structure: {structure}
-                The refactored JSON disctionary: """
-        return prompt
-    
-    def prompt_gpt_redo_v2(self, incorrect_json):
-        structure = """
-            {"Dictionary":{
-                "catalog_number": "",
-                "genus": "",
-                "species": "".
-                "subspecies": "",
-                "variety": "",
-                "forma":"",
-                "country": "",
-                "state": "",
-                "county": "",
-                "locality_name": "",
-                "min_elevation": "",
-                "max_elevation": "",
-                "elevation_units": "',
-                "verbatim_coordinates": "",
-                "decimal_coordinates": "",
-                "datum": "",
-                "cultivated": "",
-                "habitat": "",
-                "plant_description": "",
-                "collectors": "",
-                "collector_number": "",
-                "determined_by": "",
-                "multiple_names": "',
-                "verbatim_date": "",
-                "date": "",
-                "end_date": "",
-                },
-            "SpeciesName": {"taxonomy": [Genus_species]}}"""
-        
-        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
-                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
-                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-                The incorrectly formatted JSON dictionary: {incorrect_json}
-                The output JSON structure: {structure}
-                The refactored JSON disctionary: """
-        return prompt
-    #####################################################################################################################################
-    #####################################################################################################################################
-    def prompt_v1_palm2(self, in_list, out_list, OCR=None):
-        self.OCR = OCR or self.OCR
-        set_rules = """1. Your job is to return a new dict based on the structure of the reference dict ref_dict and these are your rules. 
-                        2. You must look at ref_dict and refactor the new text called OCR to match the same formatting. 
-                        3. OCR contains unstructured text inside of [], use your knowledge to put the OCR text into the correct ref_dict column. 
-                        4. If OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None" and skip all other steps.
-                        5. If there is a field that does not have a direct proxy in the OCR text, you can fill it in based on your knowledge, but you cannot generate new information.
-                        6. Never put text from the ref_dict values into the new dict, but you must use the headers from ref_dict. 
-                        7. There cannot be duplicate dictionary fields.
-                        8. Only return the new dict, do not explain your answer.
-                        9. Do not include quotation marks in content, only use quotation marks to represent values in dictionaries.
-                        10. For GPS coordinates only use Decimal Degrees (D.D°)
-                        11. "Given the input text, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values."""
-
-        umich_all_asia_rules = """
-        "Catalog Number" - {"format": "[barcode]", "null_value": "", "description": the barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits}
-        "Genus" - {"format": "[Genus]" or "[Family] indet" if no genus", "null_value": "", "description": taxonomic determination to genus, do captalize genus}
-        "Species"- {"format": "[species]" or "indet" if no species, "null_value": "", "description": taxonomic determination to species, do not captalize species}
-        "subspecies" - {"format": "[subspecies]", "null_value": "", "description": taxonomic determination to subspecies (subsp.)}
-        "variety" - {"format": "[variety]", "null_value": "", "description": taxonomic determination to variety (var)}
-        "forma" - {"format": "[form]", "null_value": "", "description": taxonomic determination to form (f.)}
-
-        "Country" - {"format": "[Country]", "null_value": "no data", "description": Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abreviation is given}
-        "State" - {"format": "[Adm. Division 1]", "null_value": "no data", "description": Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word}
-        "County" - {"format": "[Adm. Division 2]", "null_value": "no data", "description": Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word}
-        "Locality Name" - {"format": "verbatim", if no geographic info: "no data provided on label of catalog no: [######]", or if illegible: "locality present but illegible/not translated for catalog no: #######", or if no named locality: "no named locality for catalog no: #######", "description": "Description of geographic location or landscape"}
-
-        "Min Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given}
-        "Max Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise}
-        "Elevation Units" - {format: "m", "null_value": "","description": "m" only if an elevation is present}
-        
-        "Verbatim Coordinates" - {"format": "[Lat, Long | UTM | TRS]", "null_value": "", "description": Convert coordinates to Decimal Degrees (D.D°) format, do not use Minutes, Seconds or quotation marks}
-
-        "Datum" - {"format": "[WGS84, NAD23 etc.]", "null_value": "not present", "description": Datum of coordinates on label; "" is GPS coordinates are not in OCR}
-        "Cultivated" - {"format": "yes", "null_value": "", "description": Indicates if specimen was grown in cultivation}
-        "Habitat" - {"format": "verbatim", "null_value": "", "description": Description of habitat or location where specimen was collected, ignore descriptions of the plant itself}
-        "Collectors" - {"format": "[Collector]", "null_value": "not present", "description": Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas}
-        "Collector Number" - {"format": "[Collector No.]", "null_value": "s.n.", "description": Sequential number assigned to collection, associated with the collector}
-        "Verbatim Date" - {"format": "verbatim", "null_value": "s.d.", "description": Date of collection exactly as it appears on the label}
-        "Date" - {"format": "[yyyy-mm-dd]", "null_value": "", "description": Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e. 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day}
-        "End Date" - {"format": "[yyyy-mm-dd]", "null_value": "", "description": If date range is listed, later date of collection range}
-        """
-
-        prompt = f"""Given the following set of rules:
-
-                set_rules = {set_rules}
-
-                Some dict fields have special requirements listed below. First is the column header. After the - is the format. Do not include the instructions with your response:
-
-                requirements = {umich_all_asia_rules}
-
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-
-                input: {in_list[0]}
-
-                output: {out_list[0]}
-
-                input: {in_list[1]}
-
-                output: {out_list[1]}
-
-                input: {in_list[2]}
-
-                output: {out_list[2]}
-
-                input: {self.OCR}
-
-                output:"""
-        
-        return prompt
-    
-    def prompt_v1_palm2_noDomainKnowledge(self, OCR=None):
-        self.OCR = OCR or self.OCR
-        set_rules = """1. Your job is to return a new dict based on the structure of the reference dict ref_dict and these are your rules. 
-                        2. You must look at ref_dict and refactor the new text called OCR to match the same formatting. 
-                        3. OCR contains unstructured text inside of [], use your knowledge to put the OCR text into the correct ref_dict column. 
-                        4. If OCR is mostly empty and contains substantially less text than the ref_dict examples, then only return "None" and skip all other steps.
-                        5. If there is a field that does not have a direct proxy in the OCR text, you can fill it in based on your knowledge, but you cannot generate new information.
-                        6. Never put text from the ref_dict values into the new dict, but you must use the headers from ref_dict. 
-                        7. There cannot be duplicate dictionary fields.
-                        8. Only return the new dict, do not explain your answer.
-                        9. Do not include quotation marks in content, only use quotation marks to represent values in dictionaries.
-                        10. For GPS coordinates only use Decimal Degrees (D.D°)
-                        11. "Given the input text, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values."""
-
-        umich_all_asia_rules = """
-        "Catalog Number" - {"format": "barcode", "null_value": "", "description": the barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits}
-        "Genus" - {"format": "Genus" or "Family indet" if no genus", "null_value": "", "description": taxonomic determination to genus, do captalize genus}
-        "Species"- {"format": "species" or "indet" if no species, "null_value": "", "description": taxonomic determination to species, do not captalize species}
-        "subspecies" - {"format": "subspecies", "null_value": "", "description": taxonomic determination to subspecies (subsp.)}
-        "variety" - {"format": "variety", "null_value": "", "description": taxonomic determination to variety (var)}
-        "forma" - {"format": "form", "null_value": "", "description": taxonomic determination to form (f.)}
-
-        "Country" - {"format": "Country", "null_value": "no data", "description": Country that corresponds to the current geographic location of collection; capitalize first letter of each word; use the entire location name even if an abreviation is given}
-        "State" - {"format": "Adm. Division 1", "null_value": "no data", "description": Administrative division 1 that corresponds to the current geographic location of collection; capitalize first letter of each word}
-        "County" - {"format": "Adm. Division 2", "null_value": "no data", "description": Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word}
-        "Locality Name" - {"format": "verbatim", if no geographic info: "no data provided on label of catalog no: ######", or if illegible: "locality present but illegible/not translated for catalog no: #######", or if no named locality: "no named locality for catalog no: #######", "description": "Description of geographic location or landscape"}
-
-        "Min Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, default field for elevation if a range is not given}
-        "Max Elevation" - {format: "elevation integer", "null_value": "","description": Elevation or altitude in meters, convert from feet to meters if 'm' or 'meters' is not in the text and round to integer, maximum elevation if there are two elevations listed but '' otherwise}
-        "Elevation Units" - {format: "m", "null_value": "","description": "m" only if an elevation is present}
-        
-        "Verbatim Coordinates" - {"format": "Lat, Long, UTM, TRS", "null_value": "", "description": Convert coordinates to Decimal Degrees (D.D°) format, do not use Minutes, Seconds or quotation marks}
-
-        "Datum" - {"format": "WGS84, NAD23 etc.", "null_value": "not present", "description": Datum of coordinates on label; "" is GPS coordinates are not in OCR}
-        "Cultivated" - {"format": "yes", "null_value": "", "description": Indicates if specimen was grown in cultivation}
-        "Habitat" - {"format": "verbatim", "null_value": "", "description": Description of habitat or location where specimen was collected, ignore descriptions of the plant itself}
-        "Collectors" - {"format": "Collector", "null_value": "not present", "description": Full name of person (i.e., agent) who collected the specimen; if more than one person then separate the names with commas}
-        "Collector Number" - {"format": "Collector No.", "null_value": "s.n.", "description": Sequential number assigned to collection, associated with the collector}
-        "Verbatim Date" - {"format": "verbatim", "null_value": "s.d.", "description": Date of collection exactly as it appears on the label}
-        "Date" - {"format": "yyyy-mm-dd", "null_value": "", "description": Date of collection formatted as year, month, and day; zeros may be used for unknown values i.e. 0000-00-00 if no date, YYYY-00-00 if only year, YYYY-MM-00 if no day}
-        "End Date" - {"format": "yyyy-mm-dd", "null_value": "", "description": If date range is listed, later date of collection range}
-        """
-        structure = """{
-            "Catalog Number": "",
-            "Genus": "",
-            "Species": "",
-            "subspecies": "",
-            "variety": "",
-            "forma": "",
-            "Country": "",
-            "State": "",
-            "County": "",
-            "Locality Name": "",
-            "Min Elevation": "",
-            "Max Elevation": "",
-            "Elevation Units": "",
-            "Verbatim Coordinates": "",
-            "Datum": "",
-            "Cultivated": "",
-            "Habitat": "",
-            "Collectors": "",
-            "Collector Number": "",
-            "Verbatim Date": "",
-            "Date": "",
-            "End Date": "",
-            }"""
-        # structure = """{
-        #             "Catalog Number": [Catalog Number],
-        #             "Genus": [Genus],
-        #             "Species": [species],
-        #             "subspecies": [subspecies],
-        #             "variety": [variety],
-        #             "forma": [forma],
-        #             "Country": [Country],
-        #             "State": [State],
-        #             "County": [County],
-        #             "Locality Name": [Locality Name],
-        #             "Min Elevation": [Min Elevation],
-        #             "Max Elevation": [Max Elevation],
-        #             "Elevation Units": [Elevation Units],
-        #             "Verbatim Coordinates": [Verbatim Coordinates],
-        #             "Datum": [Datum],
-        #             "Cultivated": [Cultivated],
-        #             "Habitat": [Habitat],
-        #             "Collectors": [Collectors],
-        #             "Collector Number": [Collector Number],
-        #             "Verbatim Date": [Verbatim Date],
-        #             "Date": [Date],
-        #             "End Date": [End Date]
-        #             }"""
-
-        prompt = f"""Given the following set of rules:
-                set_rules = {set_rules}
-                Some dict fields have special requirements listed below. First is the column header. After the - is the format. Do not include the instructions with your response:
-                requirements = {umich_all_asia_rules}
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-                The input unformatted OCR text: {self.OCR}
-                The output JSON structure: {structure}
-                The output JSON structure: {structure}
-                The output JSON structure: {structure}
-                The refactored JSON disctionary:"""
-        
-        return prompt
-    
-    def prompt_v2_palm2(self, OCR=None):
-        self.OCR = OCR or self.OCR
-        self.n_fields = 26 or self.n_fields
-
-        set_rules = """
-        1. Refactor the unstructured OCR text into a dictionary based on the JSON structure outlined below.
-        2. You should map the unstructured OCR text to the appropriate JSON key and then populate the field based on its rules.
-        3. Some JSON key fields are permitted to remain empty if the corresponding information is not found in the unstructured OCR text.
-        4. Ignore any information in the OCR text that doesn't fit into the defined JSON structure.
-        5. Duplicate dictionary fields are not allowed.
-        6. Ensure that all JSON keys are in lowercase.
-        7. Ensure that new JSON field values follow sentence case capitalization.
-        8. Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-        9. Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-        10. Only return a JSON dictionary represented as a string. You should not explain your answer.
-        """
-
-        dictionary_field_format_descriptions = """
-        The next section of instructions outlines how to format the JSON dictionary. The keys are the same as those of the final formatted JSON object.
-        For each key there is a format requirement that specifies how to transcribe the information for that key. 
-        The possible formatting options are:
-        1. "verbatim transcription" - field is populated with verbatim text from the unformatted OCR.
-        2. "spell check transcription" - field is populated with spelling corrected text from the unformatted OCR.
-        3. "boolean yes no" - field is populated with only yes or no.
-        4. "integer" - field is populated with only an integer.
-        5. "[list]" - field is populated from one of the values in the list.
-        6. "yyyy-mm-dd" - field is populated with a date in the format year-month-day.
-        The desired null value is also given. Populate the field with the null value of the information for that key is not present in the unformatted OCR text. 
-        """
-
-        json_template_rules = """
-            {
-                "catalog_number": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "The barcode identifier, typically a number with at least 6 digits, but fewer than 30 digits."
-                },
-                "genus": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to genus. Genus must be capitalized. If genus is not present use the taxonomic family name followed by the word 'indet'."
-                },
-                "species": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to species, do not capitalize species."
-                },
-                "subspecies": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to subspecies (subsp.)."
-                },
-                "variety": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to variety (var)."
-                },
-                "forma": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Taxonomic determination to form (f.)."
-                },
-                "country": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Country that corresponds to the current geographic location of collection. Capitalize first letter of each word. If abbreviation is given populate field with the full spelling of the country's name. Use sentence-case to capitalize proper nouns."
-                },
-                "state": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Administrative division 1 that corresponds to the current geographic location of collection. Capitalize first letter of each word. Administrative division 1 is equivalent to a U.S. State. Use sentence-case to capitalize proper nouns."
-                },
-                "county": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Administrative division 2 that corresponds to the current geographic location of collection; capitalize first letter of each word. Administrative division 2 is equivalent to a U.S. county, parish, borough. Use sentence-case to capitalize proper nouns."
-                },
-                "locality_name": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Description of geographic location, landscape, landmarks, regional features, nearby places, or any contextual information aiding in pinpointing the exact origin or site of the specimen. Use sentence-case to capitalize proper nouns."
-                },
-                "min_elevation": {
-                    "format": "integer",
-                    "null_value": "",
-                    "description": "Minimum elevation or altitude in meters. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
-                },
-                "max_elevation": {
-                    "format": "integer",
-                    "null_value": "",
-                    "description": "Maximum elevation or altitude in meters. If only one elevation is present, then max_elevation should be set to the null_value. Only if units are explicit then convert from feet ('ft' or 'ft.' or 'feet') to meters ('m' or 'm.' or 'meters'). Round to integer."
-                },
-                "elevation_units": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Elevation units must be meters. If min_elevation field is populated, then elevation_units: 'm'. Otherwise elevation_units: ''"
-                },
-                "verbatim_coordinates": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Verbatim location coordinates as they appear on the label. Do not convert formats. Possible coordinate types are one of [Lat, Long, UTM, TRS]."
-                },
-                "decimal_coordinates": {
-                    "format": "spell check transcription",
-                    "null_value": "",
-                    "description": "Correct and convert the verbatim location coordinates to conform with the decimal degrees GPS coordinate format."
-                },
-                "datum": {
-                    "format": "[WGS84, WGS72, WGS66, WGS60, NAD83, NAD27, OSGB36, ETRS89, ED50, GDA94, JGD2011, Tokyo97, KGD2002, TWD67, TWD97, BJS54, XAS80, GCJ-02, BD-09, PZ-90.11, GTRF, CGCS2000, ITRF88, ITRF89, ITRF90, ITRF91, ITRF92, ITRF93, ITRF94, ITRF96, ITRF97, ITRF2000, ITRF2005, ITRF2008, ITRF2014, Hong Kong Principal Datum, SAD69]",
-                    "null_value": "",
-                    "description": "Datum of location coordinates. Possible values are include in the format list. Leave field blank if unclear."
-                },
-                "cultivated": {
-                    "format": "boolean yes no",
-                    "null_value": "",
-                    "description": "Cultivated plants are intentionally grown by humans. In text descriptions, look for planting dates, garden locations, ornamental, cultivar names, garden, or farm to indicate cultivated plant."
-                },
-                "habitat": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Description of a plant's habitat or the location where the specimen was collected. Ignore descriptions of the plant itself. Use sentence-case to capitalize proper nouns."
-                },
-                "plant_description": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Description of plant features such as leaf shape, size, color, stem texture, height, flower structure, scent, fruit or seed characteristics, root system type, overall growth habit and form, any notable aroma or secretions, presence of hairs or bristles, and any other distinguishing morphological or physiological characteristics. Use sentence-case to capitalize proper nouns."
-                },
-                "collectors": {
-                    "format": "verbatim transcription",
-                    "null_value": "not present",
-                    "description": "Full name(s) of the individual(s) responsible for collecting the specimen. Use sentence-case to capitalize proper nouns. When multiple collectors are involved, their names should be separated by commas."
-                },
-                "collector_number": {
-                    "format": "verbatim transcription",
-                    "null_value": "s.n.",
-                    "description": "Unique identifier or number that denotes the specific collecting event and associated with the collector."
-                },
-                "determined_by": {
-                    "format": "verbatim transcription",
-                    "null_value": "",
-                    "description": "Full name of the individual responsible for determining the taxanomic name of the specimen. Use sentence-case to capitalize proper nouns. Sometimes the name will be near to the characters 'det' to denote determination. This name may be isolated from other names in the unformatted OCR text."
-                },
-                "multiple_names": {
-                    "format": "boolean yes no",
-                    "null_value": "",
-                    "description": "Indicate whether multiple people or collector names are present in the unformatted OCR text. Use sentence-case to capitalize proper nouns. If you see more than one person's name the value is 'yes'; otherwise the value is 'no'."
-                },
-                "verbatim_date": {
-                    "format": "verbatim transcription",
-                    "null_value": "s.d.",
-                    "description": "Date of collection exactly as it appears on the label. Do not change the format or correct typos."
-                },
-                "date": {
-                    "format": "yyyy-mm-dd",
-                    "null_value": "",
-                    "description": "Date the specimen was collected formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire date is unknown, 'YYYY-00-00' if only the year is known, and 'YYYY-MM-00' if year and month are known but day is not."
-                },
-                "end_date": {
-                    "format": "yyyy-mm-dd",
-                    "null_value": "",
-                    "description": "If a date range is provided, this represents the later or ending date of the collection period, formatted as year-month-day. If specific components of the date are unknown, they should be replaced with zeros. Examples: '0000-00-00' if the entire end date is unknown, 'YYYY-00-00' if only the year of the end date is known, and 'YYYY-MM-00' if year and month of the end date are known but the day is not."
-                },
-            }"""
-   
-        structure = """{"catalog_number": "",
-                        "genus": "",
-                        "species": "".
-                        "subspecies": "",
-                        "variety": "",
-                        "forma":"",
-                        "country": "",
-                        "state": "",
-                        "county": "",
-                        "locality_name": "",
-                        "min_elevation": "",
-                        "max_elevation": "",
-                        "elevation_units": "',
-                        "verbatim_coordinates": "",
-                        "decimal_coordinates": "",
-                        "datum": "",
-                        "cultivated": "",
-                        "habitat": "",
-                        "plant_description": "",
-                        "collectors": "",
-                        "collector_number": "",
-                        "determined_by": "",
-                        "multiple_names": "',
-                        "verbatim_date": "",
-                        "date": "",
-                        "end_date": "",
-                        }"""
-        # structure = """{"catalog_number": [Catalog Number],
-        #                 "genus": [Genus],
-        #                 "species": [species],
-        #                 "subspecies": [subspecies],
-        #                 "variety": [variety],
-        #                 "forma": [forma],
-        #                 "country": [Country],
-        #                 "state": [State],
-        #                 "county": [County],
-        #                 "locality_name": [Locality Name],
-        #                 "min_elevation": [Min Elevation],
-        #                 "max_elevation": [Max Elevation],
-        #                 "elevation_units": [Elevation Units],
-        #                 "verbatim_coordinates": [Verbatim Coordinates],
-        #                 "decimal_coordinates": [Decimal Coordinates],
-        #                 "datum": [Datum],
-        #                 "cultivated": [boolean yes no],
-        #                 "habitat": [Habitat Description],
-        #                 "plant_description": [Plant Description],
-        #                 "collectors": [Name(s) of Collectors],
-        #                 "collector_number": [Collector Number],
-        #                 "determined_by": [Name(s) of Taxonomist],
-        #                 "multiple_names": [boolean yes no],
-        #                 "verbatim_date": [Verbatim Date],
-        #                 "date": [yyyy-mm-dd],
-        #                 "end_date": [yyyy-mm-dd],
-        #                 }"""
-
-        prompt = f"""Please help me complete this text parsing task given the following rules and unstructured OCR text. Your task is to refactor the OCR text into a structured JSON dictionary that matches the structure specified in the following rules. Please follow the rules strictly.
-        The rules are:
-        {set_rules}
-        The unstructured OCR text is:
-        {self.OCR}
-        {dictionary_field_format_descriptions}
-        This is the JSON template that includes instructions for each key:
-        {json_template_rules}
-        Please populate the following JSON dictionary based on the rules and the unformatted OCR text. The square brackets denote the locations that you should place the new structured text:
-        {structure}
-        {structure}
-        {structure}
-        """
-        
-        return prompt
-    
-    def prompt_palm_redo_v1(self, incorrect_json):
-        structure = """{
-                    "Catalog Number": [Catalog Number],
-                    "Genus": [Genus],
-                    "Species": [species],
-                    "subspecies": [subspecies],
-                    "variety": [variety],
-                    "forma": [forma],
-                    "Country": [Country],
-                    "State": [State],
-                    "County": [County],
-                    "Locality Name": [Locality Name],
-                    "Min Elevation": [Min Elevation],
-                    "Max Elevation": [Max Elevation],
-                    "Elevation Units": [Elevation Units],
-                    "Verbatim Coordinates": [Verbatim Coordinates],
-                    "Datum": [Datum],
-                    "Cultivated": [Cultivated],
-                    "Habitat": [Habitat],
-                    "Collectors": [Collectors],
-                    "Collector Number": [Collector Number],
-                    "Verbatim Date": [Verbatim Date],
-                    "Date": [Date],
-                    "End Date": [End Date]
-                    }"""
-
-        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
-                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
-                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-                The incorrectly formatted JSON dictionary: {incorrect_json}
-                The output JSON structure: {structure}
-                The output JSON structure: {structure}
-                The output JSON structure: {structure}
-                The refactored JSON disctionary: """
-        return prompt
-        
-    def prompt_palm_redo_v2(self, incorrect_json):
-        structure = """{"catalog_number": "",
-                        "genus": "",
-                        "species": "".
-                        "subspecies": "",
-                        "variety": "",
-                        "forma":"",
-                        "country": "",
-                        "state": "",
-                        "county": "",
-                        "locality_name": "",
-                        "min_elevation": "",
-                        "max_elevation": "",
-                        "elevation_units": "',
-                        "verbatim_coordinates": "",
-                        "decimal_coordinates": "",
-                        "datum": "",
-                        "cultivated": "",
-                        "habitat": "",
-                        "plant_description": "",
-                        "collectors": "",
-                        "collector_number": "",
-                        "determined_by": "",
-                        "multiple_names": "',
-                        "verbatim_date": "",
-                        "date": "",
-                        "end_date": "",
-                        }"""
-        
-        prompt = f"""This text is supposed to be JSON, but it contains an error that prevents it from loading with the Python command json.loads().
-                You need to return coorect JSON for the following dictionary. Most likely, a quotation mark inside of a field value has not been escaped properly with a backslash.
-                Given the input, please generate a JSON response. Please note that the response should not contain any special characters, including quotation marks (single ' or double \"), within the JSON values.
-                Escape all JSON control characters that appear in input including ampersand (&) and other control characters. 
-                Ensure all key-value pairs in the JSON dictionary strictly adhere to the format and data types specified in the template.
-                Ensure the output JSON string is valid JSON format. It should not have trailing commas or unquoted keys.
-                The incorrectly formatted JSON dictionary: {incorrect_json}
-                The output JSON structure: {structure}
-                The output JSON structure: {structure}
-                The output JSON structure: {structure}
-                The refactored JSON disctionary: """
-        return prompt

vouchervision/utils_LLM.py

@@ -0,0 +1,102 @@
+# Helper funcs for LLM_XXXXX.py
+import tiktoken, json, os
+from langchain_core.output_parsers.format_instructions import JSON_FORMAT_INSTRUCTIONS
+from transformers import AutoTokenizer
+import GPUtil
+import time
+import psutil
+import threading
+import torch
+
+
+def remove_colons_and_double_apostrophes(text):
+    return text.replace(":", "").replace("\"", "")
+
+
+
+def count_tokens(string, vendor, model_name):
+    full_string = string + JSON_FORMAT_INSTRUCTIONS
+    
+    def run_count(full_string, model_name):
+            # Ensure the encoding is obtained correctly.
+            encoding = tiktoken.encoding_for_model(model_name)
+            tokens = encoding.encode(full_string)
+            return len(tokens)
+        
+    try:
+        if vendor == 'mistral':
+            tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
+
+            tokens = tokenizer.tokenize(full_string)
+            return len(tokens)
+        
+        else:
+            return run_count(full_string, model_name)
+        
+    except Exception as e:
+        print(f"An error occurred: {e}")
+        return 0
+    
+
+
+class SystemLoadMonitor():
+    def __init__(self, logger) -> None:
+        self.monitoring_thread = None
+        self.logger = logger
+        self.gpu_usage = {'max_cpu_usage': 0, 'max_load': 0, 'max_vram_usage': 0, "max_ram_usage": 0, 'monitoring': True}
+        self.start_time = None
+        self.has_GPU = torch.cuda.is_available()
+        self.monitor_interval = 2
+        
+    def start_monitoring_usage(self):
+        self.start_time = time.time()
+        self.monitoring_thread = threading.Thread(target=self.monitor_usage, args=(self.monitor_interval,))
+        self.monitoring_thread.start()
+
+    def monitor_usage(self, interval):
+        while self.gpu_usage['monitoring']:
+            # GPU monitoring
+            if self.has_GPU:
+                GPUs = GPUtil.getGPUs()
+                for gpu in GPUs:
+                    self.gpu_usage['max_load'] = max(self.gpu_usage['max_load'], gpu.load)
+                    # Convert memory usage to GB
+                    memory_usage_gb = gpu.memoryUsed / 1024.0
+                    self.gpu_usage['max_vram_usage'] = max(self.gpu_usage.get('max_vram_usage', 0), memory_usage_gb)
+            
+            # RAM monitoring
+            ram_usage = psutil.virtual_memory().used / (1024.0 ** 3)  # Get RAM usage in GB
+            self.gpu_usage['max_ram_usage'] = max(self.gpu_usage.get('max_ram_usage', 0), ram_usage)
+            
+            # CPU monitoring
+            cpu_usage = psutil.cpu_percent(interval=None)
+            self.gpu_usage['max_cpu_usage'] = max(self.gpu_usage.get('max_cpu_usage', 0), cpu_usage)
+            time.sleep(interval)
+
+    def stop_monitoring_report_usage(self):
+        self.gpu_usage['monitoring'] = False
+        self.monitoring_thread.join()
+        elapsed_time = time.time() - self.start_time
+        self.logger.info(f"Inference Time: {round(elapsed_time,2)} seconds")
+        
+        self.logger.info(f"Max CPU Usage: {round(self.gpu_usage['max_cpu_usage'],2)}%")
+        self.logger.info(f"Max RAM Usage: {round(self.gpu_usage['max_ram_usage'],2)}GB")  
+
+        if self.has_GPU:
+          self.logger.info(f"Max GPU Load: {round(self.gpu_usage['max_load']*100,2)}%")
+          self.logger.info(f"Max GPU Memory Usage: {round(self.gpu_usage['max_vram_usage'],2)}GB")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

vouchervision/utils_LLM_JSON_validation.py

@@ -0,0 +1,170 @@
+import json
+
+
+def validate_and_align_JSON_keys_with_template(data, JSON_dict_structure):
+    data = validate_JSON(data)
+    if data is None:
+        return None
+    else:
+        # Make sure that there are no literal list [] objects in the dict
+        for key, value in data.items():
+            if value is None:
+                data[key] = ''
+            elif isinstance(value, str):
+                if value.lower() in ['unknown', 'not provided', 'missing', 'na', 'none', 'n/a', 'null', 
+                                    'not provided in the text', 'not found in the text', 
+                                    'not in the text', 'not provided', 'not found',
+                                    'not provided in the OCR', 'not found in the OCR', 
+                                    'not in the OCR', 
+                                    'not provided in the OCR text', 'not found in the OCR text', 
+                                    "not specified in the given text.",
+                                    "not specified in the given text",
+                                    "not specified in the text.",
+                                    "not specified in the text",
+                                    "not specified in text.",
+                                    "not specified in text",
+                                    "not specified in OCR",
+                                    "not specified",
+                                    'not in the OCR text', 
+                                    'n/a n/a','n/a, n/a',
+                                    'n/a, n/a, n/a','n/a n/a, n/a','n/a, n/a n/a','n/a n/a n/a',
+                                    'n/a, n/a, n/a, n/a','n/a n/a n/a n/a','n/a n/a, n/a, n/a','n/a, n/a n/a, n/a','n/a, n/a, n/a n/a',
+                                    'n/a n/a n/a, n/a','n/a, n/a n/a n/a',
+                                    'n/a n/a, n/a n/a',]:
+                    data[key] = ''
+            elif isinstance(value, list):
+                # Join the list elements into a single string
+                data[key] = ', '.join(str(item) for item in value)
+
+        ### align the keys with the template
+        # Create a new dictionary with the same order of keys as JSON_dict_structure
+        ordered_data = {}
+
+        # This will catch cases where the LLM JSON case does not match the required JSON key's case
+        for key in JSON_dict_structure:
+            truth_key_lower = key.lower()
+            
+            llm_value = str(data.get(key, ''))
+            if not llm_value:
+                llm_value = str(data.get(truth_key_lower, ''))
+            
+            # Copy the value from data if it exists, else use an empty string
+            ordered_data[key] = llm_value
+
+        return ordered_data
+
+
+
+
+def validate_JSON(data):
+    if isinstance(data, dict):
+        return data
+    else:
+        if isinstance(data, list):
+            data = data[0]
+        try:
+            json_candidate = json.loads(data) # decoding the JSON data
+            if isinstance(json_candidate, list):
+                json_candidate = json_candidate[0]
+            
+            if isinstance(json_candidate, dict):
+                data = json_candidate
+                return data
+            else:
+                return None
+        except:
+            return None  
+        
+
+
+
+#### A manual method for pulling a JSON object out of a verbose LLM response.
+#### It's messy butr works well enough. Only use if JSONparsing is not available
+def extract_json_dict_manual(text):
+    text = text.strip().replace("\n", "").replace("\\\\", "")
+    # Find the first opening curly brace
+    start_index = text.find('{')
+    # Find the last closing curly brace
+    end_index = text.rfind('}') + 1
+
+    text = text[start_index:end_index]
+
+    # Find the first opening curly brace
+    start_index = text.find('{')
+    # Find the last closing curly brace
+    end_index = text.rfind('}') + 1
+
+    # Check and remove backslash immediately after the opening curly brace
+    if text[start_index + 1] == "\\":
+        text = text[:start_index + 1] + text[start_index + 2:]
+
+    # Find the first opening curly brace
+    start_index = text.find('{')
+    # Find the last closing curly brace
+    end_index = text.rfind('}') + 1
+
+    # print(text[end_index-2])
+    if text[end_index-2] == "\\":
+        text = text[:end_index-2] + text[end_index-1]
+    else:
+        text = text
+
+    # Find the first opening curly brace
+    start_index = text.find('{')
+    # Find the last closing curly brace
+    end_index = text.rfind('}') + 1
+
+    # print(text[end_index-2])
+    if text[end_index-2] == "\\":
+        text = text[:end_index-3] + text[end_index-1]
+    else:
+        text = text
+
+    # Trim whitespace
+    json_str = text
+
+    # Print the JSON string for inspection
+    # print("Extracted JSON string:", json_str)
+
+    # Convert JSON string to Python dictionary
+    try:
+        # If necessary, replace newline characters
+        # json_str = json_str.replace('\n', '\\n')
+
+        json_dict = json.loads(json_str)
+        return json_dict
+    except Exception as e:
+        print("Error parsing JSON:", e)
+        return None
+'''
+if __name__ == "__main__":
+    tex = """Extracted JSON string: {"catalogNumber": "MPU395640",
+    "order": "Monimizeae",
+    "family": "Monimiaceae",
+    "scientificName": "Hedycarya parvifolia",
+    "scientificNameAuthorship": "Perkins & Schltr.",
+    "genus": "Hedycarya",
+    "subgenus": null,
+    "specificEpithet": "parvifolia",
+    "infraspecificEpithet": null,
+    "identifiedBy": null,
+    "recordedBy": "R. Pouteau & J. Munzinger",
+    "recordNumber": "RP 1",
+    "verbatimEventDate": "26-2-2013",
+    "eventDate": "2013-02-26",
+    "habitat": "Ultramafique Long. 165 ° 52'21  E, Lat. 21 ° 29'19  S, Maquis",
+    "occurrenceRemarks": "Fruit Arbuste, Ht. 1,5 mètre (s), Fruits verts (immatures) à noirs (matures), Coll. R. Pouteau & J. Munzinger N° RP 1, Dupl. P - MPU, RECOLNAT, Herbier IRD de Nouvelle - Calédonie Poutan 1, Golden Thread, Alt. 818 mètre, Diam. 2 cm, Date 26-2-2013",
+    "country": "Nouvelle Calédonie",
+    "stateProvince": null,
+    "county": null,
+    "municipality": null,
+    "locality": "Antenne du plateau de Bouakaine",
+    "degreeOfEstablishment": "cultivated",
+    "decimalLatitude": -21.488611,
+    "decimalLongitude": 165.8725,
+    "verbatimCoordinates": "Long. 165 ° 52'21  E, Lat. 21 ° 29'19  S",
+    "minimumElevationInMeters": 818,
+    "maximumElevationInMeters": 818
+    \\}"""
+    new = extract_json_dict(tex)
+'''

vouchervision/utils_VoucherVision.py

@@ -1,26 +1,23 @@
 import openai
-import os, sys, json, inspect, glob, tiktoken, shutil, yaml
+import os, json, glob, shutil, yaml, torch, logging
 import openpyxl
 from openpyxl import Workbook, load_workbook
-import google.generativeai as palm
-from langchain.chat_models import AzureChatOpenAI
-from google.oauth2 import service_account
-
-currentdir = os.path.dirname(os.path.abspath(
-    inspect.getfile(inspect.currentframe())))
-parentdir = os.path.dirname(currentdir)
-sys.path.append(parentdir)
-parentdir = os.path.dirname(parentdir)
-sys.path.append(parentdir)
-
-from general_utils import get_cfg_from_full_path, num_tokens_from_string
-from embeddings_db import VoucherVisionEmbedding
-from OCR_google_cloud_vision import detect_text, overlay_boxes_on_image
-from LLM_chatGPT_3_5 import OCR_to_dict, OCR_to_dict_16k
-from LLM_PaLM import OCR_to_dict_PaLM
-# from LLM_Falcon import OCR_to_dict_Falcon
-from prompts import PROMPT_UMICH_skeleton_all_asia, PROMPT_OCR_Organized, PROMPT_UMICH_skeleton_all_asia_GPT4, PROMPT_OCR_Organized_GPT4, PROMPT_JSON
-from prompt_catalog import PromptCatalog
+import vertexai
+from transformers import TrOCRProcessor, VisionEncoderDecoderModel
+from langchain_openai import AzureChatOpenAI
+from OCR_google_cloud_vision import OCRGoogle 
+
+from vouchervision.LLM_OpenAI import OpenAIHandler
+from vouchervision.LLM_GooglePalm2 import GooglePalm2Handler
+from vouchervision.LLM_GoogleGemini import GoogleGeminiHandler
+from vouchervision.LLM_MistralAI import MistralHandler
+from vouchervision.LLM_local_cpu_MistralAI import LocalCPUMistralHandler
+from vouchervision.LLM_local_MistralAI import LocalMistralHandler 
+from vouchervision.utils_LLM import remove_colons_and_double_apostrophes
+from vouchervision.prompt_catalog import PromptCatalog
+from vouchervision.model_maps import ModelMaps
+from vouchervision.general_utils import get_cfg_from_full_path
+
 '''
 * For the prefix_removal, the image names have 'MICH-V-' prior to the barcode, so that is used for matching
   but removed for output.
@@ -29,24 +26,11 @@ from prompt_catalog import PromptCatalog
         - Look for ####################### Catalog Number pre-defined
 '''
 
-'''
-Prior to StructuredOutputParser:
-    response = openai.ChatCompletion.create(
-            model=MODEL,
-            temperature = 0,
-            messages=[
-                {"role": "system", "content": "You are a helpful assistant acting as a transcription expert and your job is to transcribe herbarium specimen labels based on OCR data and reformat it to meet Darwin Core Archive Standards into a Python dictionary based on certain rules."},
-                {"role": "user", "content": prompt},
-            ],
-            max_tokens=2048,
-        )
-        # print the model's response
-        return response.choices[0].message['content']
-'''
 
+    
 class VoucherVision():
 
-    def __init__(self, cfg, logger, dir_home, path_custom_prompts, Project, Dirs):
+    def __init__(self, cfg, logger, dir_home, path_custom_prompts, Project, Dirs, is_hf):
         self.cfg = cfg
         self.logger = logger
         self.dir_home = dir_home
@@ -55,7 +39,12 @@ class VoucherVision():
         self.Dirs = Dirs
         self.headers = None
         self.prompt_version = None
-        self.client = None
+        self.is_hf = is_hf
+
+        # self.trOCR_model_version = "microsoft/trocr-large-handwritten"
+        self.trOCR_model_version = "microsoft/trocr-base-handwritten"
+        self.trOCR_processor = None
+        self.trOCR_model = None
 
         self.set_API_keys()
         self.setup()
@@ -78,15 +67,25 @@ class VoucherVision():
         self.prompt_version0 = self.cfg['leafmachine']['project']['prompt_version']
         self.use_domain_knowledge = self.cfg['leafmachine']['project']['use_domain_knowledge']
 
-        self.catalog_name_options = ["Catalog Number", "catalog_number"]
+        self.catalog_name_options = ["Catalog Number", "catalog_number", "catalogNumber"]
+
+        self.utility_headers = ["filename",
+                                "WFO_override_OCR", "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_candidate_names","WFO_placement",
+                                
+                                "GEO_override_OCR", "GEO_method", "GEO_formatted_full_string", "GEO_decimal_lat",
+                                "GEO_decimal_long","GEO_city", "GEO_county", "GEO_state",
+                                "GEO_state_code", "GEO_country", "GEO_country_code", "GEO_continent",
+                                
+                                "tokens_in", "tokens_out", "path_to_crop","path_to_original","path_to_content","path_to_helper",]
 
-        self.utility_headers = ["tokens_in", "tokens_out", "path_to_crop","path_to_original","path_to_content","path_to_helper",]
+        self.do_create_OCR_helper_image = self.cfg['leafmachine']['do_create_OCR_helper_image']
 
         self.map_prompt_versions()
         self.map_dir_labels()
         self.map_API_options()
-        self.init_embeddings()
+        # self.init_embeddings()
         self.init_transcription_xlsx()
+        self.init_trOCR_model()
 
         '''Logging'''
         self.logger.info(f'Transcribing dataset --- {self.dir_labels}')
@@ -98,41 +97,31 @@ class VoucherVision():
         self.logger.info(f'     Model name passed to API --> {self.model_name}')
         self.logger.info(f'     API access token is found in PRIVATE_DATA.yaml --> {self.has_key}')
 
-    # def map_API_options(self):
-    #     self.chat_version = self.cfg['leafmachine']['LLM_version']
-    #     version_mapping = {
-    #         'GPT 4': ('OpenAI GPT 4', False, 'GPT_4', self.has_key_openai),
-    #         'GPT 3.5': ('OpenAI GPT 3.5', False, 'GPT_3_5', self.has_key_openai),
-    #         'Azure GPT 3.5': ('(Azure) OpenAI GPT 3.5', True, 'Azure_GPT_3_5', self.has_key_azure_openai),
-    #         'Azure GPT 4': ('(Azure) OpenAI GPT 4', True, 'Azure_GPT_4', self.has_key_azure_openai),
-    #         'PaLM 2': ('Google PaLM 2', None, None, self.has_key_palm2)
-    #     }
-    #     if self.chat_version not in version_mapping:
-    #         supported_LLMs = ", ".join(version_mapping.keys())
-    #         raise Exception(f"Unsupported LLM: {self.chat_version}. Requires one of: {supported_LLMs}")
-
-    #     self.version_name, self.is_azure, self.model_name, self.has_key = version_mapping[self.chat_version]
+    def init_trOCR_model(self):
+        lgr = logging.getLogger('transformers')
+        lgr.setLevel(logging.ERROR)
+        
+        self.trOCR_processor = TrOCRProcessor.from_pretrained(self.trOCR_model_version)
+        self.trOCR_model = VisionEncoderDecoderModel.from_pretrained(self.trOCR_model_version)
+        
+        # Check for GPU availability
+        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+        self.trOCR_model.to(self.device)
+
     def map_API_options(self):
-        self.chat_version = self.cfg['leafmachine']['LLM_version']  # Replace with your env variable for LLM version if needed
-
-        # Assuming you have set your environment variables for each key like 'OPENAI_API_KEY', 'AZURE_API_KEY', 'PALM_API_KEY'
-        openai_api_key = os.getenv('OPENAI_API_KEY')
-        azure_api_key = os.getenv('AZURE_API_KEY')
-        palm_api_key = os.getenv('PALM_API_KEY')
-
-        version_mapping = {
-            'GPT 4': ('OpenAI GPT 4', False, 'GPT_4', bool(openai_api_key)),
-            'GPT 3.5': ('OpenAI GPT 3.5', False, 'GPT_3_5', bool(openai_api_key)),
-            'Azure GPT 3.5': ('(Azure) OpenAI GPT 3.5', True, 'Azure_GPT_3_5', bool(azure_api_key)),
-            'Azure GPT 4': ('(Azure) OpenAI GPT 4', True, 'Azure_GPT_4', bool(azure_api_key)),
-            'PaLM 2': ('Google PaLM 2', None, None, bool(palm_api_key))
-        }
+        self.chat_version = self.cfg['leafmachine']['LLM_version']
 
-        if self.chat_version not in version_mapping:
-            supported_llms = ", ".join(version_mapping.keys())
-            raise Exception(f"Unsupported LLM: {self.chat_version}. Requires one of: {supported_llms}")
+        # Get the required values from ModelMaps
+        self.model_name = ModelMaps.get_version_mapping_cost(self.chat_version)
+        self.is_azure = ModelMaps.get_version_mapping_is_azure(self.chat_version)
+        self.has_key = ModelMaps.get_version_has_key(self.chat_version, self.has_key_openai, self.has_key_azure_openai, self.has_key_palm2, self.has_key_mistral)
 
-        self.version_name, self.is_azure, self.model_name, self.has_key = version_mapping[self.chat_version]
+        # Check if the version is supported
+        if self.model_name is None:
+            supported_LLMs = ", ".join(ModelMaps.get_models_gui_list())
+            raise Exception(f"Unsupported LLM: {self.chat_version}. Requires one of: {supported_LLMs}")
+
+        self.version_name = self.chat_version
 
     def map_prompt_versions(self):
         self.prompt_version_map = {
@@ -149,13 +138,13 @@ class VoucherVision():
     def is_in_prompt_version_map(self, value):
         return value in self.prompt_version_map.values()
 
-    def init_embeddings(self):
-        if self.use_domain_knowledge:
-            self.logger.info(f'*** USING DOMAIN KNOWLEDGE ***')
-            self.logger.info(f'*** Initializing vector embeddings database ***')
-            self.initialize_embeddings()
-        else:
-            self.Voucher_Vision_Embedding = None
+    # def init_embeddings(self):
+    #     if self.use_domain_knowledge:
+    #         self.logger.info(f'*** USING DOMAIN KNOWLEDGE ***')
+    #         self.logger.info(f'*** Initializing vector embeddings database ***')
+    #         self.initialize_embeddings()
+    #     else:
+    #         self.Voucher_Vision_Embedding = None
 
     def map_dir_labels(self):
         if self.cfg['leafmachine']['use_RGB_label_images']:
@@ -176,53 +165,41 @@ class VoucherVision():
             
     def generate_xlsx_headers(self):
         # Extract headers from the 'Dictionary' keys in the JSON template rules
-        xlsx_headers = list(self.rules_config_json['rules']["Dictionary"].keys())
+        # xlsx_headers = list(self.rules_config_json['rules']["Dictionary"].keys())
+        xlsx_headers = list(self.rules_config_json['rules'].keys())
         xlsx_headers = xlsx_headers + self.utility_headers
         return xlsx_headers
 
     def init_transcription_xlsx(self):
-        self.HEADERS_v1_n22 = ["Catalog Number","Genus","Species","subspecies","variety","forma","Country","State","County","Locality Name","Min Elevation","Max Elevation","Elevation Units","Verbatim Coordinates","Datum","Cultivated","Habitat","Collectors","Collector Number","Verbatim Date","Date","End Date"] 
-        self.HEADERS_v2_n26 = ["catalog_number","genus","species","subspecies","variety","forma","country","state","county","locality_name","min_elevation","max_elevation","elevation_units","verbatim_coordinates","decimal_coordinates","datum","cultivated","habitat","plant_description","collectors","collector_number","determined_by","multiple_names","verbatim_date","date","end_date"]
-        self.HEADERS_v1_n22 = self.HEADERS_v1_n22 + self.utility_headers
-        self.HEADERS_v2_n26 = self.HEADERS_v2_n26 + self.utility_headers
+        # self.HEADERS_v1_n22 = ["Catalog Number","Genus","Species","subspecies","variety","forma","Country","State","County","Locality Name","Min Elevation","Max Elevation","Elevation Units","Verbatim Coordinates","Datum","Cultivated","Habitat","Collectors","Collector Number","Verbatim Date","Date","End Date"] 
+        # self.HEADERS_v2_n26 = ["catalog_number","genus","species","subspecies","variety","forma","country","state","county","locality_name","min_elevation","max_elevation","elevation_units","verbatim_coordinates","decimal_coordinates","datum","cultivated","habitat","plant_description","collectors","collector_number","determined_by","multiple_names","verbatim_date","date","end_date"]
+        # self.HEADERS_v1_n22 = self.HEADERS_v1_n22 + self.utility_headers
+        # self.HEADERS_v2_n26 = self.HEADERS_v2_n26 + self.utility_headers
         # Initialize output file
         self.path_transcription = os.path.join(self.Dirs.transcription,"transcribed.xlsx")
 
-        if self.prompt_version in ['prompt_v2_json_rules','prompt_v2_palm2']:
-            self.headers = self.HEADERS_v2_n26
-            self.headers_used = 'HEADERS_v2_n26'
+        # if self.prompt_version in ['prompt_v2_json_rules','prompt_v2_palm2']:
+            # self.headers = self.HEADERS_v2_n26
+            # self.headers_used = 'HEADERS_v2_n26'
         
-        elif self.prompt_version in ['prompt_v1_verbose', 'prompt_v1_verbose_noDomainKnowledge','prompt_v1_palm2', 'prompt_v1_palm2_noDomainKnowledge']:
-            self.headers = self.HEADERS_v1_n22
-            self.headers_used = 'HEADERS_v1_n22'
+        # elif self.prompt_version in ['prompt_v1_verbose', 'prompt_v1_verbose_noDomainKnowledge','prompt_v1_palm2', 'prompt_v1_palm2_noDomainKnowledge']:
+            # self.headers = self.HEADERS_v1_n22
+            # self.headers_used = 'HEADERS_v1_n22'
         
+        # else:
+        if not self.is_predefined_prompt:
+            # Load the rules configuration
+            self.rules_config_json = self.load_rules_config()
+            # Generate the headers from the configuration
+            self.headers = self.generate_xlsx_headers()
+            # Set the headers used to the dynamically generated headers
+            self.headers_used = 'CUSTOM'
         else:
-            if not self.is_predefined_prompt:
-                # Load the rules configuration
-                self.rules_config_json = self.load_rules_config()
-                # Generate the headers from the configuration
-                self.headers = self.generate_xlsx_headers()
-                # Set the headers used to the dynamically generated headers
-                self.headers_used = 'CUSTOM'
-            else:
-                # If it's a predefined prompt, raise an exception as we don't have further instructions
-                raise ValueError("Predefined prompt is not handled in this context.")
+            # If it's a predefined prompt, raise an exception as we don't have further instructions
+            raise ValueError("Predefined prompt is not handled in this context.")
 
         self.create_or_load_excel_with_headers(os.path.join(self.Dirs.transcription,"transcribed.xlsx"), self.headers)
 
-
-    def pick_model(self, vendor, nt):
-        if vendor == 'GPT_3_5':
-            if nt > 6000:
-                return "gpt-3.5-turbo-16k-0613", True
-            else:
-                return "gpt-3.5-turbo", False
-        if vendor == 'GPT_4':
-            return "gpt-4", False
-        if vendor == 'Azure_GPT_3_5':
-            return "gpt-35-turbo", False
-        if vendor == 'Azure_GPT_4':
-            return "gpt-4", False
            
     def create_or_load_excel_with_headers(self, file_path, headers, show_head=False):
         output_dir_names = ['Archival_Components', 'Config_File', 'Cropped_Images', 'Logs', 'Original_Images', 'Transcription']
@@ -324,7 +301,14 @@ class VoucherVision():
 
 
 
-    def add_data_to_excel_from_response(self, path_transcription, response, filename_without_extension, path_to_crop, path_to_content, path_to_helper, nt_in, nt_out):
+    def add_data_to_excel_from_response(self, path_transcription, response, WFO_record, GEO_record, filename_without_extension, path_to_crop, path_to_content, path_to_helper, nt_in, nt_out):
+        geo_headers = ["GEO_override_OCR", "GEO_method", "GEO_formatted_full_string", "GEO_decimal_lat",
+                       "GEO_decimal_long","GEO_city", "GEO_county", "GEO_state",
+                       "GEO_state_code", "GEO_country", "GEO_country_code", "GEO_continent",]
+        
+        # WFO_candidate_names is separate, bc it may be type --> list
+        wfo_headers = ["WFO_override_OCR", "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_placement"]
+
         wb = openpyxl.load_workbook(path_transcription)
         sheet = wb.active
 
@@ -385,68 +369,182 @@ class VoucherVision():
                 sheet.cell(row=next_row, column=i, value=nt_in)
             elif header.value == "tokens_out":
                 sheet.cell(row=next_row, column=i, value=nt_out)
+            elif header.value == "filename":
+                sheet.cell(row=next_row, column=i, value=filename_without_extension)
+
+            # "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_candidate_names","WFO_placement"
+            elif header.value in wfo_headers:
+                sheet.cell(row=next_row, column=i, value=WFO_record.get(header.value, ''))
+            # elif header.value == "WFO_exact_match":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_exact_match",''))
+            # elif header.value == "WFO_exact_match_name":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_exact_match_name",''))
+            # elif header.value == "WFO_best_match":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_best_match",''))
+            # elif header.value == "WFO_placement":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_placement",''))
+            elif header.value == "WFO_candidate_names":
+                candidate_names = WFO_record.get("WFO_candidate_names", '')
+                # Check if candidate_names is a list and convert to a string if it is
+                if isinstance(candidate_names, list):
+                    candidate_names_str = '|'.join(candidate_names)
+                else:
+                    candidate_names_str = candidate_names
+                sheet.cell(row=next_row, column=i, value=candidate_names_str)
+            
+            # "GEO_method", "GEO_formatted_full_string", "GEO_decimal_lat", "GEO_decimal_long",
+            # "GEO_city", "GEO_county", "GEO_state", "GEO_state_code", "GEO_country", "GEO_country_code", "GEO_continent"
+            elif header.value in geo_headers:
+                sheet.cell(row=next_row, column=i, value=GEO_record.get(header.value, ''))
+
         # save the workbook
         wb.save(path_transcription)
+    
 
-    def configure_azure_llm(self):
-        # Access the secrets from the environment
-        azure_api_version = os.getenv('AZURE_API_VERSION')
-        azure_api_key = os.getenv('AZURE_API_KEY')
-        azure_api_base = os.getenv('AZURE_API_BASE')
-        azure_organization = os.getenv('AZURE_ORGANIZATION')
-        azure_api_type = os.getenv('AZURE_API_TYPE')
-        azure_deployment_name = os.getenv('AZURE_DEPLOYMENT_NAME')
-
-        # Check if all required Azure configurations are present
-        if azure_api_version and azure_api_key and azure_api_base and azure_organization and azure_api_type and azure_deployment_name:
-            self.llm = AzureChatOpenAI(
-                deployment_name=azure_deployment_name,
-                openai_api_version=azure_api_version,
-                openai_api_key=azure_api_key,
-                openai_api_base=azure_api_base,
-                openai_organization=azure_organization,
-                openai_api_type=azure_api_type
-            )
+    def has_API_key(self, val):
+        if val != '':
+            return True
         else:
-            raise ValueError("Missing Azure configuration in environment variables.")
+            return False
+        
+    
+
+        
+    
+    
 
     def set_API_keys(self):
-        # Access secrets directly from the environment
-        openai_api_key = os.getenv('OPENAI_API_KEY')
-        google_application_credentials = os.getenv('GOOGLE_APPLICATION_CREDENTIALS')
-        palm_api_key = os.getenv('PALM_API_KEY')
-
-        self.has_key_openai = openai_api_key is not None
-        self.has_key_google_OCR = google_application_credentials is not None
-        self.has_key_palm2 = palm_api_key is not None
-
-        if self.has_key_google_OCR:
-            # Get the credentials JSON from the environment variable
-            google_credentials_json = os.getenv('GOOGLE_APPLICATION_CREDENTIALS')
-            # Convert the JSON string into a Python dictionary
-            google_credentials_dict = json.loads(google_credentials_json)
-            # Create a credentials object
-            google_credentials = service_account.Credentials.from_service_account_info(google_credentials_dict)
-
-            # Now, use this `google_credentials` object to authenticate your Google Cloud services
-            # For example, if you are using the Google Vision API, it would look like this:
-            from google.cloud import vision
-            self.client = vision.ImageAnnotatorClient(credentials=google_credentials)
-
-        if os.getenv('AZURE_API_KEY') is not None:
-            self.configure_azure_llm()
-
-        if self.has_key_palm2:
-            palm.configure(api_key=palm_api_key)
-
-        if self.has_key_openai:
-            openai.api_key = openai_api_key
-            os.environ["OPENAI_API_KEY"] = openai_api_key
+        if self.is_hf:
+            openai_api_key = os.getenv('OPENAI_API_KEY')
+            google_application_credentials = os.getenv('GOOGLE_APPLICATION_CREDENTIALS')
+            palm_api_key = os.getenv('PALM_API_KEY')
+            mistral_api_key = os.getenv('MISTRAL_API_KEY')
+            here_api_key = os.getenv('here_api_key')
+            here_app_id = os.getenv('here_app_id')
+            open_cage_api_key = os.getenv('open_cage_geocode')
+            google_project_id = os.getenv('GOOGLE_PROJECT_ID')
+            google_project_location = os.getenv('GOOGLE_LOCATION')
+
+
+
+            self.has_key_openai = openai_api_key is not None
+            self.has_key_google_OCR = google_application_credentials is not None
+            self.has_key_palm2 = palm_api_key is not None
+            self.has_key_mistral = mistral_api_key is not None
+            self.has_key_here = here_api_key is not None
+            self.has_hey_here = here_app_id is not None
+            self.has_open_cage_geocode = open_cage_api_key is not None
+            self.has_key_google_project_id = google_project_id is not None
+            self.has_key_google_project_location = google_project_location is not None
+
+            if self.has_key_openai:
+                openai.api_key = openai_api_key
+
+            if self.has_key_google_project_id and self.has_key_google_project_location:
+                vertexai.init(project=os.getenv('GOOGLE_PROJECT_ID'), location=os.getenv('GOOGLE_LOCATION'))
+
+            if os.getenv('AZURE_API_KEY') is not None:
+                azure_api_version = os.getenv('AZURE_API_VERSION')
+                azure_api_key = os.getenv('AZURE_API_KEY')
+                azure_api_base = os.getenv('AZURE_API_BASE')
+                azure_organization = os.getenv('AZURE_ORGANIZATION')
+                azure_api_type = os.getenv('AZURE_API_TYPE')
+                azure_deployment_name = os.getenv('AZURE_DEPLOYMENT_NAME')
+
+                if azure_api_version and azure_api_key and azure_api_base and azure_organization and azure_api_type and azure_deployment_name:
+
+                    self.has_key_azure_openai = True
+                    self.llm = AzureChatOpenAI(
+                        deployment_name = 'gpt-35-turbo',#'gpt-35-turbo',
+                        openai_api_version = azure_api_version,
+                        openai_api_key = azure_api_key,
+                        azure_endpoint = azure_api_base,
+                        openai_organization = azure_organization,
+                    )
+
+
+        else:
+            self.dir_home = os.path.dirname(os.path.dirname(__file__))
+            self.path_cfg_private = os.path.join(self.dir_home, 'PRIVATE_DATA.yaml')
+            self.cfg_private = get_cfg_from_full_path(self.path_cfg_private)
+
+            self.has_key_openai = self.has_API_key(self.cfg_private['openai']['OPENAI_API_KEY'])
+
+            self.has_key_azure_openai = self.has_API_key(self.cfg_private['openai_azure']['api_version'])
+            
+            self.has_key_google_OCR = self.has_API_key(self.cfg_private['google_cloud']['path_json_file'])
+
+            self.has_key_palm2 = self.has_API_key(self.cfg_private['google_palm']['google_palm_api'])
+            self.has_key_google_project_id = self.has_API_key(self.cfg_private['google_palm']['project_id'])
+            self.has_key_google_project_location = self.has_API_key(self.cfg_private['google_palm']['location'])
+
+            self.has_key_mistral = self.has_API_key(self.cfg_private['mistral']['mistral_key'])
+
+            self.has_key_here = self.has_API_key(self.cfg_private['here']['api_key'])
+
+            self.has_open_cage_geocode = self.has_API_key(self.cfg_private['open_cage_geocode']['api_key'])
+
+            if self.has_key_openai:
+                openai.api_key = self.cfg_private['openai']['OPENAI_API_KEY']
+                os.environ["OPENAI_API_KEY"] = self.cfg_private['openai']['OPENAI_API_KEY']
+                
+            if self.has_key_azure_openai:
+                # os.environ["OPENAI_API_KEY"] = self.cfg_private['openai_azure']['openai_api_key']
+                self.llm = AzureChatOpenAI(
+                    deployment_name = 'gpt-35-turbo',#'gpt-35-turbo',
+                    openai_api_version = self.cfg_private['openai_azure']['api_version'],
+                    openai_api_key = self.cfg_private['openai_azure']['openai_api_key'],
+                    azure_endpoint = self.cfg_private['openai_azure']['openai_api_base'],
+                    # openai_api_base=self.cfg_private['openai_azure']['openai_api_base'],
+                    openai_organization = self.cfg_private['openai_azure']['openai_organization'],
+                    # openai_api_type = self.cfg_private['openai_azure']['openai_api_type']
+                )
+            
+            # This is frustrating. a #TODO is to figure out when/why these methods conflict with the permissions set in the Palm/Gemini calls
+            name_check = self.cfg['leafmachine']['LLM_version'].lower().split(' ')
+            if ('google' in name_check) or( 'palm' in name_check) or ('gemini' in name_check):
+                os.environ['GOOGLE_PROJECT_ID'] = self.cfg_private['google_palm']['project_id'] # gemini
+                os.environ['GOOGLE_LOCATION'] = self.cfg_private['google_palm']['location'] # gemini
+                # genai.configure(api_key=self.cfg_private['google_palm']['google_palm_api'])
+                vertexai.init(project=os.environ['GOOGLE_PROJECT_ID'], location=os.environ['GOOGLE_LOCATION'])
+                # os.environ.pop("GOOGLE_APPLICATION_CREDENTIALS", None)
+                # os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.cfg_private['google_cloud']['path_json_file'] #### 
+                # os.environ['GOOGLE_API_KEY'] = self.cfg_private['google_palm']['google_palm_api']
+
+
+            ##### NOTE: this is how you can use ONLY OCR. If you get a vertexAI login it should work without loading all this
+            # else:
+            #     if self.has_key_google_OCR:
+            #         if os.path.exists(self.cfg_private['google_cloud']['path_json_file']):
+            #             os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.cfg_private['google_cloud']['path_json_file']
+            #         elif os.path.exists(self.cfg_private['google_cloud']['path_json_file_service_account2']):
+            #             os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.cfg_private['google_cloud']['path_json_file_service_account2']
+            #         else:
+            #             raise f"Google JSON API key file not found"
+
+            ##### NOTE: This should also be covered by vertexAI now
+            # if self.has_key_palm2:
+            #     os.environ['PALM'] = self.cfg_private['google_palm']['google_palm_api']
+            #     os.environ['GOOGLE_PROJECT_ID'] = self.cfg_private['google_palm']['project_id'] # gemini
+            #     os.environ['GOOGLE_LOCATION'] = self.cfg_private['google_palm']['location'] # gemini
+            #     os.environ['GOOGLE_API_KEY'] = self.cfg_private['google_palm']['google_palm_api']
+
+            if self.has_key_mistral:
+                os.environ['MISTRAL_API_KEY'] = self.cfg_private['mistral']['mistral_key']
+
+            if self.has_key_here:
+                os.environ['here_app_id'] = self.cfg_private['here']['app_id']
+                os.environ['here_api_key'] = self.cfg_private['here']['api_key']
+
+            if self.has_open_cage_geocode:
+                os.environ['open_cage_geocode'] = self.cfg_private['open_cage_geocode']['api_key']
+            
+
         
+    # def initialize_embeddings(self):
+    #     '''Loading embedding search       __init__(self, db_name, path_domain_knowledge, logger, build_new_db=False, model_name="hkunlp/instructor-xl", device="cuda")'''
+    #     self.Voucher_Vision_Embedding = VoucherVisionEmbedding(self.db_name, self.path_domain_knowledge, logger=self.logger, build_new_db=self.build_new_db)
 
-    def initialize_embeddings(self):
-        '''Loading embedding search       __init__(self, db_name, path_domain_knowledge, logger, build_new_db=False, model_name="hkunlp/instructor-xl", device="cuda")'''
-        self.Voucher_Vision_Embedding = VoucherVisionEmbedding(self.db_name, self.path_domain_knowledge, logger=self.logger, build_new_db=self.build_new_db)
 
     def clean_catalog_number(self, data, filename_without_extension):
         #Cleans up the catalog number in data if it's a dict
@@ -470,7 +568,7 @@ class VoucherVision():
             if self.headers_used == 'HEADERS_v1_n22':
                 return modify_catalog_key("Catalog Number", filename_without_extension, data)
             elif self.headers_used in ['HEADERS_v2_n26', 'CUSTOM']:
-                return modify_catalog_key("catalog_number", filename_without_extension, data)
+                return modify_catalog_key("filename", filename_without_extension, data)
             else:
                 raise ValueError("Invalid headers used.")
         else:
@@ -484,23 +582,19 @@ class VoucherVision():
                 data = json.dumps(data, indent=4, sort_keys=False)
             txt_file.write(data)
 
-    def create_null_json(self):
-        return {}
+
+    # def create_null_json(self):
+    #     return {}
     
+
     def remove_non_numbers(self, s):
         return ''.join([char for char in s if char.isdigit()])
     
+
     def create_null_row(self, filename_without_extension, path_to_crop, path_to_content, path_to_helper):
         json_dict = {header: '' for header in self.headers} 
         for header, value in json_dict.items():
-            if header in self.catalog_name_options:
-                if self.prefix_removal:
-                    json_dict[header] = filename_without_extension.replace(self.prefix_removal, "")
-                if self.suffix_removal:
-                    json_dict[header] = filename_without_extension.replace(self.suffix_removal, "")
-                if self.catalog_numerical_only:
-                    json_dict[header] = self.remove_non_numbers(json_dict[header])
-            elif header == "path_to_crop":
+            if header == "path_to_crop":
                 json_dict[header] = path_to_crop
             elif header == "path_to_original":
                 fname = os.path.basename(path_to_crop)
@@ -511,231 +605,231 @@ class VoucherVision():
                 json_dict[header] = path_to_content
             elif header == "path_to_helper":
                 json_dict[header] = path_to_helper
+            elif header == "filename":
+                json_dict[header] = filename_without_extension
+
+            # "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_candidate_names","WFO_placement"
+            elif header == "WFO_exact_match":
+                json_dict[header] =''
+            elif header == "WFO_exact_match_name":
+                json_dict[header] = ''
+            elif header == "WFO_best_match":
+                json_dict[header] = ''
+            elif header == "WFO_candidate_names":
+                json_dict[header] = ''
+            elif header == "WFO_placement":
+                json_dict[header] = ''
         return json_dict
+    
 
+    ##################################################################################################################################
+    ##################################################     OCR      ##################################################################
+    ##################################################################################################################################
+    def perform_OCR_and_save_results(self, image_index, jpg_file_path_OCR_helper, txt_file_path_OCR, txt_file_path_OCR_bounds):
+        self.logger.info(f'Working on {image_index + 1}/{len(self.img_paths)} --- Starting OCR')
+        # self.OCR - None
 
-    def setup_GPT(self, prompt_version, gpt):
-        Catalog = PromptCatalog()
-        self.logger.info(f'Length of OCR raw -- {len(self.OCR)}')
+        ### Process_image() runs the OCR for text, handwriting, trOCR AND creates the overlay image
+        ocr_google = OCRGoogle(self.path_to_crop, self.cfg, self.trOCR_model_version, self.trOCR_model, self.trOCR_processor, self.device)  
+        ocr_google.process_image(self.do_create_OCR_helper_image, self.logger)
+        self.OCR = ocr_google.OCR
 
-        # if prompt_version == 'prompt_v1_verbose':
-        if self.is_predefined_prompt:
-            if self.use_domain_knowledge:
-                # Find a similar example from the domain knowledge
-                domain_knowledge_example = self.Voucher_Vision_Embedding.query_db(self.OCR, 1)
-                similarity= self.Voucher_Vision_Embedding.get_similarity()
+        self.write_json_to_file(txt_file_path_OCR, ocr_google.OCR_JSON_to_file)
+        
+        self.logger.info(f'Working on {image_index + 1}/{len(self.img_paths)} --- Finished OCR')
 
-                if prompt_version == 'prompt_v1_verbose':
-                    prompt, n_fields, xlsx_headers = Catalog.prompt_v1_verbose(OCR=self.OCR,domain_knowledge_example=domain_knowledge_example,similarity=similarity)
+        if len(self.OCR) > 0:
+            ocr_google.overlay_image.save(jpg_file_path_OCR_helper)
 
-            else:
-                if prompt_version == 'prompt_v1_verbose_noDomainKnowledge':
-                    prompt, n_fields, xlsx_headers = Catalog.prompt_v1_verbose_noDomainKnowledge(OCR=self.OCR)
+            OCR_bounds = {}
+            if ocr_google.hand_text_to_box_mapping is not None:
+                OCR_bounds['OCR_bounds_handwritten'] = ocr_google.hand_text_to_box_mapping
 
-                elif prompt_version ==  'prompt_v2_json_rules':
-                    prompt, n_fields, xlsx_headers = Catalog.prompt_v2_json_rules(OCR=self.OCR)
-        else:
-            prompt, n_fields, xlsx_headers = Catalog.prompt_v2_custom(self.path_custom_prompts, OCR=self.OCR)
-            
+            if ocr_google.normal_text_to_box_mapping is not None:
+                OCR_bounds['OCR_bounds_printed'] = ocr_google.normal_text_to_box_mapping
 
+            if ocr_google.trOCR_text_to_box_mapping is not None:
+                OCR_bounds['OCR_bounds_trOCR'] = ocr_google.trOCR_text_to_box_mapping
 
-        nt = num_tokens_from_string(prompt, "cl100k_base")
-        self.logger.info(f'Prompt token length --- {nt}')
+            self.write_json_to_file(txt_file_path_OCR_bounds, OCR_bounds)
+            self.logger.info(f'Working on {image_index + 1}/{len(self.img_paths)} --- Saved OCR Overlay Image')
+        else:
+            pass ########################################################################################################################### fix logic for no OCR
 
-        MODEL, use_long_form = self.pick_model(gpt, nt)
-        self.logger.info(f'Waiting for {gpt} API call --- Using {MODEL}')
+    ##################################################################################################################################
+    #######################################################  LLM Switchboard  ########################################################
+    ##################################################################################################################################
+    def send_to_LLM(self, is_azure, progress_report, json_report, model_name):
+        self.n_failed_LLM_calls = 0
+        self.n_failed_OCR = 0
 
-        return MODEL, prompt, use_long_form, n_fields, xlsx_headers, nt
+        final_JSON_response = None
+        final_WFO_record = None
+        final_GEO_record = None
 
+        self.initialize_token_counters()
+        self.update_progress_report_initial(progress_report)
+
+        MODEL_NAME_FORMATTED = ModelMaps.get_API_name(model_name)
+        name_parts = model_name.split("_")
+        
+        self.setup_JSON_dict_structure()
         
-    # def setup_GPT(self, opt, gpt):
-    #     if opt == 'dict':
-    #         # Find a similar example from the domain knowledge
-    #         domain_knowledge_example = self.Voucher_Vision_Embedding.query_db(self.OCR, 1)
-    #         similarity= self.Voucher_Vision_Embedding.get_similarity()
+        json_report.set_text(text_main=f'Loading {MODEL_NAME_FORMATTED}')
+        json_report.set_JSON({}, {}, {})
+        llm_model = self.initialize_llm_model(self.logger, MODEL_NAME_FORMATTED, self.JSON_dict_structure, name_parts, is_azure, self.llm)
 
-    #         self.logger.info(f'Length of OCR raw -- {len(self.OCR)}')
+        for i, path_to_crop in enumerate(self.img_paths):
+            self.update_progress_report_batch(progress_report, i)
 
-    #         # prompt = PROMPT_UMICH_skeleton_all_asia_GPT4(self.OCR, domain_knowledge_example, similarity)
-    #         prompt, n_fields, xlsx_headers = 
+            if self.should_skip_specimen(path_to_crop):
+                self.log_skipping_specimen(path_to_crop)
+                continue
 
-    #         nt = num_tokens_from_string(prompt, "cl100k_base")
-    #         self.logger.info(f'Prompt token length --- {nt}')
+            paths = self.generate_paths(path_to_crop, i)
+            self.path_to_crop = path_to_crop
 
-    #         MODEL, use_long_form = self.pick_model(gpt, nt)
+            filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper = paths
+            json_report.set_text(text_main='Starting OCR')
+            self.perform_OCR_and_save_results(i, jpg_file_path_OCR_helper, txt_file_path_OCR, txt_file_path_OCR_bounds)
+            json_report.set_text(text_main='Finished OCR')
 
-    #         ### Direct GPT ###
-    #         self.logger.info(f'Waiting for {MODEL} API call --- Using chatGPT --- Content')
+            if not self.OCR:
+                self.n_failed_OCR += 1
+                response_candidate = None
+                nt_in = 0
+                nt_out = 0
+            else:
+                ### Format prompt
+                prompt = self.setup_prompt()
+                prompt = remove_colons_and_double_apostrophes(prompt)
 
-    #         return MODEL, prompt, use_long_form
-        
-    #     elif opt == 'helper':
-    #         prompt = PROMPT_OCR_Organized_GPT4(self.OCR)
-    #         nt = num_tokens_from_string(prompt, "cl100k_base")
+                ### Send prompt to chosen LLM
+                self.logger.info(f'Waiting for {model_name} API call --- Using {MODEL_NAME_FORMATTED}')
 
-    #         MODEL, use_long_form = self.pick_model(gpt, nt)
+                if 'PALM2' in name_parts:
+                    response_candidate, nt_in, nt_out, WFO_record, GEO_record = llm_model.call_llm_api_GooglePalm2(prompt, json_report)
+                
+                elif 'GEMINI' in name_parts:
+                    response_candidate, nt_in, nt_out, WFO_record, GEO_record = llm_model.call_llm_api_GoogleGemini(prompt, json_report)
+                
+                elif 'MISTRAL' in name_parts and ('LOCAL' not in name_parts):
+                    response_candidate, nt_in, nt_out, WFO_record, GEO_record = llm_model.call_llm_api_MistralAI(prompt, json_report)
+                
+                elif 'LOCAL' in name_parts: 
+                    if 'MISTRAL' in name_parts or 'MIXTRAL' in name_parts:
+                        if 'CPU' in name_parts:     
+                            response_candidate, nt_in, nt_out, WFO_record, GEO_record = llm_model.call_llm_local_cpu_MistralAI(prompt, json_report) 
+                        else:
+                            response_candidate, nt_in, nt_out, WFO_record, GEO_record = llm_model.call_llm_local_MistralAI(prompt, json_report) 
+                else:
+                    response_candidate, nt_in, nt_out, WFO_record, GEO_record = llm_model.call_llm_api_OpenAI(prompt, json_report)
 
-    #         self.logger.info(f'Length of OCR raw -- {len(self.OCR)}')
-    #         self.logger.info(f'Prompt token length --- {nt}')
-    #         self.logger.info(f'Waiting for {MODEL} API call --- Using chatGPT --- Helper')
+            self.n_failed_LLM_calls += 1 if response_candidate is None else 0
+                
+            ### Estimate n tokens returned
+            self.logger.info(f'Prompt tokens IN --- {nt_in}')
+            self.logger.info(f'Prompt tokens OUT --- {nt_out}')
+                
+            self.update_token_counters(nt_in, nt_out)
 
-    #         return MODEL, prompt, use_long_form
+            final_JSON_response, final_WFO_record, final_GEO_record = self.update_final_response(response_candidate, WFO_record, GEO_record, paths, path_to_crop, nt_in, nt_out)
 
+            self.log_completion_info(final_JSON_response)
 
-    def use_chatGPT(self, is_azure, progress_report, gpt):
-        total_tokens_in = 0
-        total_tokens_out = 0
-        final_JSON_response = None
-        if progress_report is not None:
-            progress_report.set_n_batches(len(self.img_paths))
-        for i, path_to_crop in enumerate(self.img_paths):
-            if progress_report is not None:
-                progress_report.update_batch(f"Working on image {i+1} of {len(self.img_paths)}")
+            json_report.set_JSON(final_JSON_response, final_WFO_record, final_GEO_record)
+
+        self.update_progress_report_final(progress_report)
+        final_JSON_response = self.parse_final_json_response(final_JSON_response)
+        return final_JSON_response, final_WFO_record, final_GEO_record, self.total_tokens_in, self.total_tokens_out
+    
 
-            if os.path.basename(path_to_crop) in self.completed_specimens:
-                self.logger.info(f'[Skipping] specimen {os.path.basename(path_to_crop)} already processed')
+    ##################################################################################################################################
+    ################################################## LLM Helper Funcs ##############################################################
+    ##################################################################################################################################
+    def initialize_llm_model(self, logger, model_name, JSON_dict_structure, name_parts, is_azure=None, llm_object=None):
+        if 'LOCAL'in name_parts:
+            if ('MIXTRAL' in name_parts) or ('MISTRAL' in name_parts):
+                if 'CPU' in name_parts:
+                    return LocalCPUMistralHandler(logger, model_name, JSON_dict_structure)
+                else:
+                    return LocalMistralHandler(logger, model_name, JSON_dict_structure)
+        else:
+            if 'PALM2' in name_parts:
+                return GooglePalm2Handler(logger, model_name, JSON_dict_structure)
+            elif 'GEMINI' in name_parts:
+                return GoogleGeminiHandler(logger, model_name, JSON_dict_structure)
+            elif 'MISTRAL' in name_parts and ('LOCAL' not in name_parts):
+                return MistralHandler(logger, model_name, JSON_dict_structure)
             else:
-                filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper = self.generate_paths(path_to_crop, i)
-
-                # Use Google Vision API to get OCR
-                # self.OCR = detect_text(path_to_crop) 
-                self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Starting OCR')
-                self.OCR, self.bounds, self.text_to_box_mapping = detect_text(path_to_crop, self.client)
-                self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Finished OCR')
-                if len(self.OCR) > 0:
-                    self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Creating OCR Overlay Image')
-                    self.overlay_image = overlay_boxes_on_image(path_to_crop, self.bounds, self.cfg['leafmachine']['do_create_OCR_helper_image'])
-                    self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Saved OCR Overlay Image')
-                    
-                    self.write_json_to_file(txt_file_path_OCR, {"OCR":self.OCR})
-                    self.write_json_to_file(txt_file_path_OCR_bounds, {"OCR_Bounds":self.bounds})
-                    self.overlay_image.save(jpg_file_path_OCR_helper)
-
-                    # Setup Dict
-                    MODEL, prompt, use_long_form, n_fields, xlsx_headers, nt_in = self.setup_GPT(self.prompt_version, gpt)
-
-                    if is_azure:
-                        self.llm.deployment_name = MODEL
-                    else:
-                        self.llm = None
-
-                    # Send OCR to chatGPT and return formatted dictonary
-                    if use_long_form:
-                        response_candidate = OCR_to_dict_16k(is_azure, self.logger, MODEL, prompt, self.llm, self.prompt_version) 
-                        nt_out = num_tokens_from_string(response_candidate, "cl100k_base")
-                    else:
-                        response_candidate = OCR_to_dict(is_azure, self.logger, MODEL, prompt, self.llm, self.prompt_version)
-                        nt_out = num_tokens_from_string(response_candidate, "cl100k_base")
-                else: 
-                    response_candidate = None
-                    nt_out = 0
-
-                total_tokens_in += nt_in
-                total_tokens_out += nt_out
-
-                final_JSON_response0 = self.save_json_and_xlsx(response_candidate, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
-                if response_candidate is not None:
-                    final_JSON_response = final_JSON_response0
-
-                self.logger.info(f'Formatted JSON\n{final_JSON_response}')
-                self.logger.info(f'Finished {MODEL} API calls\n')
-        
-        if progress_report is not None:
-            progress_report.reset_batch(f"Batch Complete")
-        try:
-            final_JSON_response = json.loads(final_JSON_response.strip('```').replace('json\n', '', 1).replace('json', '', 1))
-        except:
-            pass
-        return final_JSON_response, total_tokens_in, total_tokens_out
+                return OpenAIHandler(logger, model_name, JSON_dict_structure, is_azure, llm_object)
+
+    def setup_prompt(self):
+        Catalog = PromptCatalog()
+        prompt, _ = Catalog.prompt_SLTP(self.path_custom_prompts, OCR=self.OCR)
+        return prompt
+    
+    def setup_JSON_dict_structure(self):
+        Catalog = PromptCatalog()
+        _, self.JSON_dict_structure = Catalog.prompt_SLTP(self.path_custom_prompts, OCR='Text')
+    
 
-                    
+    def initialize_token_counters(self):
+        self.total_tokens_in = 0
+        self.total_tokens_out = 0
 
-    def use_PaLM(self, progress_report):
-        total_tokens_in = 0
-        total_tokens_out = 0
-        final_JSON_response = None
+
+    def update_progress_report_initial(self, progress_report):
         if progress_report is not None:
             progress_report.set_n_batches(len(self.img_paths))
-        for i, path_to_crop in enumerate(self.img_paths):
-            if progress_report is not None:
-                progress_report.update_batch(f"Working on image {i+1} of {len(self.img_paths)}")
-            if os.path.basename(path_to_crop) in self.completed_specimens:
-                self.logger.info(f'[Skipping] specimen {os.path.basename(path_to_crop)} already processed')
-            else:
-                filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper = self.generate_paths(path_to_crop, i)
-                
-                # Use Google Vision API to get OCR
-                self.OCR, self.bounds, self.text_to_box_mapping = detect_text(path_to_crop, self.client)
-                if len(self.OCR) > 0:
-                    self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Starting OCR')
-                    self.OCR = self.OCR.replace("\'", "Minutes").replace('\"', "Seconds")
-                    self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Finished OCR')
-
-                    self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Creating OCR Overlay Image')
-                    self.overlay_image = overlay_boxes_on_image(path_to_crop, self.bounds, self.cfg['leafmachine']['do_create_OCR_helper_image'])
-                    self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- Saved OCR Overlay Image')
-                    
-                    self.write_json_to_file(txt_file_path_OCR, {"OCR":self.OCR})
-                    self.write_json_to_file(txt_file_path_OCR_bounds, {"OCR_Bounds":self.bounds})
-                    self.overlay_image.save(jpg_file_path_OCR_helper)
-
-                    # Send OCR to chatGPT and return formatted dictonary
-                    response_candidate, nt_in = OCR_to_dict_PaLM(self.logger, self.OCR, self.prompt_version, self.Voucher_Vision_Embedding)
-                    nt_out = num_tokens_from_string(response_candidate, "cl100k_base")
-                    
-                else:
-                    response_candidate = None
-                    nt_out = 0
-
-                total_tokens_in += nt_in
-                total_tokens_out += nt_out
 
-                final_JSON_response0 = self.save_json_and_xlsx(response_candidate, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
-                if response_candidate is not None:
-                    final_JSON_response = final_JSON_response0
-                self.logger.info(f'Formatted JSON\n{final_JSON_response}')
-                self.logger.info(f'Finished PaLM 2 API calls\n')
 
+    def update_progress_report_batch(self, progress_report, batch_index):
         if progress_report is not None:
-            progress_report.reset_batch(f"Batch Complete")
-        return final_JSON_response, total_tokens_in, total_tokens_out
+            progress_report.update_batch(f"Working on image {batch_index + 1} of {len(self.img_paths)}")
 
 
-    '''
-    def use_falcon(self, progress_report):
-        for i, path_to_crop in enumerate(self.img_paths):
-            progress_report.update_batch(f"Working on image {i+1} of {len(self.img_paths)}")
-            if os.path.basename(path_to_crop) in self.completed_specimens:
-                self.logger.info(f'[Skipping] specimen {os.path.basename(path_to_crop)} already processed')
-            else:
-                filename_without_extension = os.path.splitext(os.path.basename(path_to_crop))[0]
-                txt_file_path = os.path.join(self.Dirs.transcription_ind, filename_without_extension + '.json')
-                txt_file_path_helper = os.path.join(self.Dirs.transcription_ind_helper, filename_without_extension + '.json')
-                self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- {filename_without_extension}')
-
-                # Use Google Vision API to get OCR
-                self.OCR, self.bounds, self.text_to_box_mapping = detect_text(path_to_crop)
-                if len(self.OCR) > 0:
-                    self.OCR = self.OCR.replace("\'", "Minutes").replace('\"', "Seconds")
-
-                    # Send OCR to Falcon and return formatted dictionary
-                    response = OCR_to_dict_Falcon(self.logger, self.OCR, self.Voucher_Vision_Embedding)
-                    # response_helper = OCR_to_helper_Falcon(self.logger, OCR) # Assuming you have a similar helper function for Falcon
-                    response_helper = None
-                    
-                    self.logger.info(f'Finished Falcon API calls\n')
-                else:
-                    response = None
+    def should_skip_specimen(self, path_to_crop):
+        return os.path.basename(path_to_crop) in self.completed_specimens
+
+
+    def log_skipping_specimen(self, path_to_crop):
+        self.logger.info(f'[Skipping] specimen {os.path.basename(path_to_crop)} already processed')
 
-                if (response is not None) and (response_helper is not None):
-                    # Save transcriptions to json files
-                    self.write_json_to_file(txt_file_path, response)
-                    # self.write_json_to_file(txt_file_path_helper, response_helper)
+    
+    def update_token_counters(self, nt_in, nt_out):
+        self.total_tokens_in += nt_in
+        self.total_tokens_out += nt_out
+
+
+    def update_final_response(self, response_candidate, WFO_record, GEO_record, paths, path_to_crop, nt_in, nt_out):
+        filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper = paths
+        # Saving the JSON and XLSX files with the response and updating the final JSON response
+        if response_candidate is not None:
+            final_JSON_response_updated = self.save_json_and_xlsx(response_candidate, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
+            return final_JSON_response_updated, WFO_record, GEO_record
+        else:
+            final_JSON_response_updated = self.save_json_and_xlsx(response_candidate, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
+            return final_JSON_response_updated, WFO_record, GEO_record
 
-                    # add to the xlsx file
-                    self.add_data_to_excel_from_response(self.path_transcription, response, filename_without_extension, path_to_crop, txt_file_path, txt_file_path_helper)
-        progress_report.reset_batch()
-    '''
+
+    def log_completion_info(self, final_JSON_response):
+        self.logger.info(f'Formatted JSON\n{final_JSON_response}')
+        self.logger.info(f'Finished API calls\n')
+
+
+    def update_progress_report_final(self, progress_report):
+        if progress_report is not None:
+            progress_report.reset_batch("Batch Complete")
+
+
+    def parse_final_json_response(self, final_JSON_response):
+        try:
+            return json.loads(final_JSON_response.strip('```').replace('json\n', '', 1).replace('json', '', 1))
+        except:
+            return final_JSON_response
+    
+    
 
     def generate_paths(self, path_to_crop, i):
         filename_without_extension = os.path.splitext(os.path.basename(path_to_crop))[0]
@@ -748,49 +842,58 @@ class VoucherVision():
 
         return filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper
 
-    def save_json_and_xlsx(self, response, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out):
+
+    def save_json_and_xlsx(self, response, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out):
         if response is None:
-            response = self.create_null_json()
+            response = self.JSON_dict_structure
+            # Insert 'filename' as the first key
+            response = {'filename': filename_without_extension, **{k: v for k, v in response.items() if k != 'filename'}}
             self.write_json_to_file(txt_file_path, response)
 
             # Then add the null info to the spreadsheet
             response_null = self.create_null_row(filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper)
-            self.add_data_to_excel_from_response(self.path_transcription, response_null, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in=0, nt_out=0)
+            self.add_data_to_excel_from_response(self.path_transcription, response_null, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in=0, nt_out=0)
         
         ### Set completed JSON
         else:
             response = self.clean_catalog_number(response, filename_without_extension)
             self.write_json_to_file(txt_file_path, response)
             # add to the xlsx file
-            self.add_data_to_excel_from_response(self.path_transcription, response, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
+            self.add_data_to_excel_from_response(self.path_transcription, response, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
         return response
     
-    def process_specimen_batch(self, progress_report, is_real_run=False):
-        if is_real_run:
-            progress_report.update_overall(f"Transcribing Labels") 
+
+    def process_specimen_batch(self, progress_report, json_report, is_real_run=False):
+        if not self.has_key:
+            self.logger.error(f'No API key found for {self.version_name}')
+            raise Exception(f"No API key found for {self.version_name}")
+
         try:
-            if self.has_key:
-                if self.model_name:
-                    final_json_response, total_tokens_in, total_tokens_out = self.use_chatGPT(self.is_azure, progress_report, self.model_name)
-                else:
-                    final_json_response, total_tokens_in, total_tokens_out = self.use_PaLM(progress_report)
-                return final_json_response, total_tokens_in, total_tokens_out
-            else:
-                self.logger.info(f'No API key found for {self.version_name}')
-                raise Exception(f"No API key found for {self.version_name}")
-        except:
+            if is_real_run:
+                progress_report.update_overall(f"Transcribing Labels")
+
+            final_json_response, final_WFO_record, final_GEO_record, total_tokens_in, total_tokens_out = self.send_to_LLM(self.is_azure, progress_report, json_report, self.model_name)
+            
+            return final_json_response, final_WFO_record, final_GEO_record, total_tokens_in, total_tokens_out
+
+        except Exception as e:
+            self.logger.error(f"LLM call failed in process_specimen_batch: {e}")
             if progress_report is not None:
                 progress_report.reset_batch(f"Batch Failed")
-            self.logger.error("LLM call failed. Ending batch. process_specimen_batch()")
-            for handler in self.logger.handlers[:]:
-                handler.close()
-                self.logger.removeHandler(handler)
+            self.close_logger_handlers()
             raise
 
+
+    def close_logger_handlers(self):
+        for handler in self.logger.handlers[:]:
+            handler.close()
+            self.logger.removeHandler(handler)
+
+
     def process_specimen_batch_OCR_test(self, path_to_crop):
         for img_filename in os.listdir(path_to_crop):
             img_path = os.path.join(path_to_crop, img_filename)
-        self.OCR, self.bounds, self.text_to_box_mapping = detect_text(img_path, self.client)
+        self.OCR, self.bounds, self.text_to_box_mapping = detect_text(img_path)
 
 
 

vouchervision/utils_VoucherVision_batch.py

@@ -0,0 +1,863 @@
+import openai
+import os, sys, json, inspect, glob, tiktoken, shutil, yaml, torch, logging
+import openpyxl
+from openpyxl import Workbook, load_workbook
+import google.generativeai as genai
+import vertexai
+from langchain_openai import AzureChatOpenAI
+from transformers import TrOCRProcessor, VisionEncoderDecoderModel
+
+currentdir = os.path.dirname(os.path.abspath(
+    inspect.getfile(inspect.currentframe())))
+parentdir = os.path.dirname(currentdir)
+sys.path.append(parentdir)
+parentdir = os.path.dirname(parentdir)
+sys.path.append(parentdir)
+
+from general_utils import get_cfg_from_full_path
+# from embeddings_db import VoucherVisionEmbedding
+from OCR_google_cloud_vision import OCRGoogle 
+
+from vouchervision.LLM_OpenAI import OpenAIHandler
+from vouchervision.LLM_GooglePalm2 import GooglePalm2Handler
+from vouchervision.LLM_GoogleGemini import GoogleGeminiHandler
+from vouchervision.LLM_MistralAI import MistralHandler
+from vouchervision.LLM_local_cpu_MistralAI import LocalCPUMistralHandler
+from vouchervision.LLM_local_MistralAI import LocalMistralHandler #call_llm_local_MistralAI_8x7b
+from vouchervision.utils_LLM import remove_colons_and_double_apostrophes
+
+# from LLM_Falcon import OCR_to_dict_Falcon
+from prompt_catalog import PromptCatalog
+from vouchervision.model_maps import ModelMaps
+
+'''
+* For the prefix_removal, the image names have 'MICH-V-' prior to the barcode, so that is used for matching
+  but removed for output.
+* There is also code active to replace the LLM-predicted "Catalog Number" with the correct number since it is known.
+  The LLMs to usually assign the barcode to the correct field, but it's not needed since it is already known.
+        - Look for ####################### Catalog Number pre-defined
+'''
+
+
+    
+class VoucherVision():
+
+    def __init__(self, cfg, logger, dir_home, path_custom_prompts, Project, Dirs):
+        self.cfg = cfg
+        self.logger = logger
+        self.dir_home = dir_home
+        self.path_custom_prompts = path_custom_prompts
+        self.Project = Project
+        self.Dirs = Dirs
+        self.headers = None
+        self.prompt_version = None
+
+        # self.trOCR_model_version = "microsoft/trocr-large-handwritten"
+        self.trOCR_model_version = "microsoft/trocr-base-handwritten"
+        self.trOCR_processor = None
+        self.trOCR_model = None
+
+        self.set_API_keys()
+        self.setup()
+
+
+    def setup(self):
+        self.logger.name = f'[Transcription]'
+        self.logger.info(f'Setting up OCR and LLM')
+
+        self.db_name = self.cfg['leafmachine']['project']['embeddings_database_name']
+        self.path_domain_knowledge = self.cfg['leafmachine']['project']['path_to_domain_knowledge_xlsx']
+        self.build_new_db = self.cfg['leafmachine']['project']['build_new_embeddings_database']
+
+        self.continue_run_from_partial_xlsx = self.cfg['leafmachine']['project']['continue_run_from_partial_xlsx']
+
+        self.prefix_removal = self.cfg['leafmachine']['project']['prefix_removal']
+        self.suffix_removal = self.cfg['leafmachine']['project']['suffix_removal']
+        self.catalog_numerical_only = self.cfg['leafmachine']['project']['catalog_numerical_only']
+
+        self.prompt_version0 = self.cfg['leafmachine']['project']['prompt_version']
+        self.use_domain_knowledge = self.cfg['leafmachine']['project']['use_domain_knowledge']
+
+        self.catalog_name_options = ["Catalog Number", "catalog_number"]
+
+        self.utility_headers = ["filename",
+                                "WFO_override_OCR", "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_candidate_names","WFO_placement",
+                                
+                                "GEO_override_OCR", "GEO_method", "GEO_formatted_full_string", "GEO_decimal_lat",
+                                "GEO_decimal_long","GEO_city", "GEO_county", "GEO_state",
+                                "GEO_state_code", "GEO_country", "GEO_country_code", "GEO_continent",
+                                
+                                "tokens_in", "tokens_out", "path_to_crop","path_to_original","path_to_content","path_to_helper",]
+
+        self.do_create_OCR_helper_image = self.cfg['leafmachine']['do_create_OCR_helper_image']
+
+        self.map_prompt_versions()
+        self.map_dir_labels()
+        self.map_API_options()
+        # self.init_embeddings()
+        self.init_transcription_xlsx()
+        self.init_trOCR_model()
+
+        '''Logging'''
+        self.logger.info(f'Transcribing dataset --- {self.dir_labels}')
+        self.logger.info(f'Saving transcription batch to --- {self.path_transcription}')
+        self.logger.info(f'Saving individual transcription files to --- {self.Dirs.transcription_ind}')
+        self.logger.info(f'Starting transcription...')
+        self.logger.info(f'     LLM MODEL --> {self.version_name}')
+        self.logger.info(f'     Using Azure API --> {self.is_azure}')
+        self.logger.info(f'     Model name passed to API --> {self.model_name}')
+        self.logger.info(f'     API access token is found in PRIVATE_DATA.yaml --> {self.has_key}')
+
+    def init_trOCR_model(self):
+        lgr = logging.getLogger('transformers')
+        lgr.setLevel(logging.ERROR)
+        
+        self.trOCR_processor = TrOCRProcessor.from_pretrained(self.trOCR_model_version)
+        self.trOCR_model = VisionEncoderDecoderModel.from_pretrained(self.trOCR_model_version)
+        
+        # Check for GPU availability
+        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+        self.trOCR_model.to(self.device)
+
+    def map_API_options(self):
+        self.chat_version = self.cfg['leafmachine']['LLM_version']
+
+        # Get the required values from ModelMaps
+        self.model_name = ModelMaps.get_version_mapping_cost(self.chat_version)
+        self.is_azure = ModelMaps.get_version_mapping_is_azure(self.chat_version)
+        self.has_key = ModelMaps.get_version_has_key(self.chat_version, self.has_key_openai, self.has_key_azure_openai, self.has_key_palm2, self.has_key_mistral)
+
+        # Check if the version is supported
+        if self.model_name is None:
+            supported_LLMs = ", ".join(ModelMaps.get_models_gui_list())
+            raise Exception(f"Unsupported LLM: {self.chat_version}. Requires one of: {supported_LLMs}")
+
+        self.version_name = self.chat_version
+
+    def map_prompt_versions(self):
+        self.prompt_version_map = {
+            "Version 1": "prompt_v1_verbose",
+            "Version 1 No Domain Knowledge": "prompt_v1_verbose_noDomainKnowledge",
+            "Version 2": "prompt_v2_json_rules",
+            "Version 1 PaLM 2": 'prompt_v1_palm2',
+            "Version 1 PaLM 2 No Domain Knowledge": 'prompt_v1_palm2_noDomainKnowledge', 
+            "Version 2 PaLM 2": 'prompt_v2_palm2',
+        }
+        self.prompt_version = self.prompt_version_map.get(self.prompt_version0, self.path_custom_prompts)
+        self.is_predefined_prompt = self.is_in_prompt_version_map(self.prompt_version)
+
+    def is_in_prompt_version_map(self, value):
+        return value in self.prompt_version_map.values()
+
+    # def init_embeddings(self):
+    #     if self.use_domain_knowledge:
+    #         self.logger.info(f'*** USING DOMAIN KNOWLEDGE ***')
+    #         self.logger.info(f'*** Initializing vector embeddings database ***')
+    #         self.initialize_embeddings()
+    #     else:
+    #         self.Voucher_Vision_Embedding = None
+
+    def map_dir_labels(self):
+        if self.cfg['leafmachine']['use_RGB_label_images']:
+            self.dir_labels = os.path.join(self.Dirs.save_per_annotation_class,'label')
+        else:
+            self.dir_labels = self.Dirs.save_original
+
+        # Use glob to get all image paths in the directory
+        self.img_paths = glob.glob(os.path.join(self.dir_labels, "*"))
+
+    def load_rules_config(self):
+        with open(self.path_custom_prompts, 'r') as stream:
+            try:
+                return yaml.safe_load(stream)
+            except yaml.YAMLError as exc:
+                print(exc)
+                return None
+            
+    def generate_xlsx_headers(self):
+        # Extract headers from the 'Dictionary' keys in the JSON template rules
+        # xlsx_headers = list(self.rules_config_json['rules']["Dictionary"].keys())
+        xlsx_headers = list(self.rules_config_json['rules'].keys())
+        xlsx_headers = xlsx_headers + self.utility_headers
+        return xlsx_headers
+
+    def init_transcription_xlsx(self):
+        self.HEADERS_v1_n22 = ["Catalog Number","Genus","Species","subspecies","variety","forma","Country","State","County","Locality Name","Min Elevation","Max Elevation","Elevation Units","Verbatim Coordinates","Datum","Cultivated","Habitat","Collectors","Collector Number","Verbatim Date","Date","End Date"] 
+        self.HEADERS_v2_n26 = ["catalog_number","genus","species","subspecies","variety","forma","country","state","county","locality_name","min_elevation","max_elevation","elevation_units","verbatim_coordinates","decimal_coordinates","datum","cultivated","habitat","plant_description","collectors","collector_number","determined_by","multiple_names","verbatim_date","date","end_date"]
+        self.HEADERS_v1_n22 = self.HEADERS_v1_n22 + self.utility_headers
+        self.HEADERS_v2_n26 = self.HEADERS_v2_n26 + self.utility_headers
+        # Initialize output file
+        self.path_transcription = os.path.join(self.Dirs.transcription,"transcribed.xlsx")
+
+        if self.prompt_version in ['prompt_v2_json_rules','prompt_v2_palm2']:
+            self.headers = self.HEADERS_v2_n26
+            self.headers_used = 'HEADERS_v2_n26'
+        
+        elif self.prompt_version in ['prompt_v1_verbose', 'prompt_v1_verbose_noDomainKnowledge','prompt_v1_palm2', 'prompt_v1_palm2_noDomainKnowledge']:
+            self.headers = self.HEADERS_v1_n22
+            self.headers_used = 'HEADERS_v1_n22'
+        
+        else:
+            if not self.is_predefined_prompt:
+                # Load the rules configuration
+                self.rules_config_json = self.load_rules_config()
+                # Generate the headers from the configuration
+                self.headers = self.generate_xlsx_headers()
+                # Set the headers used to the dynamically generated headers
+                self.headers_used = 'CUSTOM'
+            else:
+                # If it's a predefined prompt, raise an exception as we don't have further instructions
+                raise ValueError("Predefined prompt is not handled in this context.")
+
+        self.create_or_load_excel_with_headers(os.path.join(self.Dirs.transcription,"transcribed.xlsx"), self.headers)
+
+           
+    def create_or_load_excel_with_headers(self, file_path, headers, show_head=False):
+        output_dir_names = ['Archival_Components', 'Config_File', 'Cropped_Images', 'Logs', 'Original_Images', 'Transcription']
+        self.completed_specimens = []
+
+        # Check if the file exists and it's not None
+        if self.continue_run_from_partial_xlsx is not None and os.path.isfile(self.continue_run_from_partial_xlsx):
+            workbook = load_workbook(filename=self.continue_run_from_partial_xlsx)
+            sheet = workbook.active
+            show_head=True
+            # Identify the 'path_to_crop' column
+            try:
+                path_to_crop_col = headers.index('path_to_crop') + 1
+                path_to_original_col = headers.index('path_to_original') + 1
+                path_to_content_col = headers.index('path_to_content') + 1
+                path_to_helper_col = headers.index('path_to_helper') + 1
+                # self.completed_specimens = list(sheet.iter_cols(min_col=path_to_crop_col, max_col=path_to_crop_col, values_only=True, min_row=2))
+            except ValueError:
+                print("'path_to_crop' not found in the header row.")
+
+            
+            path_to_crop = list(sheet.iter_cols(min_col=path_to_crop_col, max_col=path_to_crop_col, values_only=True, min_row=2))
+            path_to_original = list(sheet.iter_cols(min_col=path_to_original_col, max_col=path_to_original_col, values_only=True, min_row=2))
+            path_to_content = list(sheet.iter_cols(min_col=path_to_content_col, max_col=path_to_content_col, values_only=True, min_row=2))
+            path_to_helper = list(sheet.iter_cols(min_col=path_to_helper_col, max_col=path_to_helper_col, values_only=True, min_row=2))
+            others = [path_to_crop_col, path_to_original_col, path_to_content_col, path_to_helper_col]
+            jsons = [path_to_content_col, path_to_helper_col]
+
+            for cell in path_to_crop[0]:
+                old_path = cell
+                new_path = file_path
+                for dir_name in output_dir_names:
+                    if dir_name in old_path:
+                        old_path_parts = old_path.split(dir_name)
+                        new_path_parts = new_path.split('Transcription')
+                        updated_path = new_path_parts[0] + dir_name + old_path_parts[1]
+                        self.completed_specimens.append(os.path.basename(updated_path))
+            print(f"{len(self.completed_specimens)} images are already completed")
+
+            ### Copy the JSON files over
+            for colu in jsons:
+                cell = next(sheet.iter_rows(min_row=2, min_col=colu, max_col=colu))[0]
+                old_path = cell.value
+                new_path = file_path
+
+                old_path_parts = old_path.split('Transcription')
+                new_path_parts = new_path.split('Transcription')
+                updated_path = new_path_parts[0] + 'Transcription' + old_path_parts[1]
+
+                # Copy files
+                old_dir = os.path.dirname(old_path)
+                new_dir = os.path.dirname(updated_path)
+
+                # Check if old_dir exists and it's a directory
+                if os.path.exists(old_dir) and os.path.isdir(old_dir):
+                    # Check if new_dir exists. If not, create it.
+                    if not os.path.exists(new_dir):
+                        os.makedirs(new_dir)
+
+                    # Iterate through all files in old_dir and copy each to new_dir
+                    for filename in os.listdir(old_dir):
+                        shutil.copy2(os.path.join(old_dir, filename), new_dir) # copy2 preserves metadata
+
+            ### Update the file names
+            for colu in others:
+                for row in sheet.iter_rows(min_row=2, min_col=colu, max_col=colu):
+                    for cell in row:
+                        old_path = cell.value
+                        new_path = file_path
+                        for dir_name in output_dir_names:
+                            if dir_name in old_path:
+                                old_path_parts = old_path.split(dir_name)
+                                new_path_parts = new_path.split('Transcription')
+                                updated_path = new_path_parts[0] + dir_name + old_path_parts[1]
+                                cell.value = updated_path
+            show_head=True
+
+                
+        else:
+            # Create a new workbook and select the active worksheet
+            workbook = Workbook()
+            sheet = workbook.active
+
+            # Write headers in the first row
+            for i, header in enumerate(headers, start=1):
+                sheet.cell(row=1, column=i, value=header)
+            self.completed_specimens = []
+            
+        # Save the workbook
+        workbook.save(file_path)
+
+        if show_head:
+            print("continue_run_from_partial_xlsx:")
+            for i, row in enumerate(sheet.iter_rows(values_only=True)):
+                print(row)
+                if i == 3:  # print the first 5 rows (0-indexed)
+                    print("\n")
+                    break
+
+
+
+    def add_data_to_excel_from_response(self, path_transcription, response, WFO_record, GEO_record, filename_without_extension, path_to_crop, path_to_content, path_to_helper, nt_in, nt_out):
+        geo_headers = ["GEO_override_OCR", "GEO_method", "GEO_formatted_full_string", "GEO_decimal_lat",
+                       "GEO_decimal_long","GEO_city", "GEO_county", "GEO_state",
+                       "GEO_state_code", "GEO_country", "GEO_country_code", "GEO_continent",]
+        
+        # WFO_candidate_names is separate, bc it may be type --> list
+        wfo_headers = ["WFO_override_OCR", "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_placement"]
+
+        wb = openpyxl.load_workbook(path_transcription)
+        sheet = wb.active
+
+        # find the next empty row
+        next_row = sheet.max_row + 1
+
+        if isinstance(response, str):
+            try:
+                response = json.loads(response)
+            except json.JSONDecodeError:
+                print(f"Failed to parse response: {response}")
+                return
+
+        # iterate over headers in the first row
+        for i, header in enumerate(sheet[1], start=1):
+            # check if header value is in response keys
+            if (header.value in response) and (header.value not in self.catalog_name_options): ####################### Catalog Number pre-defined
+                # check if the response value is a dictionary
+                if isinstance(response[header.value], dict):
+                    # if it is a dictionary, extract the 'value' field
+                    cell_value = response[header.value].get('value', '')
+                else:
+                    # if it's not a dictionary, use it directly
+                    cell_value = response[header.value]
+                
+                try:
+                    # write the value to the cell
+                    sheet.cell(row=next_row, column=i, value=cell_value)
+                except:
+                    sheet.cell(row=next_row, column=i, value=cell_value[0])
+
+            elif header.value in self.catalog_name_options: 
+                # if self.prefix_removal:
+                #     filename_without_extension = filename_without_extension.replace(self.prefix_removal, "")
+                # if self.suffix_removal:
+                #     filename_without_extension = filename_without_extension.replace(self.suffix_removal, "")
+                # if self.catalog_numerical_only:
+                #     filename_without_extension = self.remove_non_numbers(filename_without_extension)
+                sheet.cell(row=next_row, column=i, value=filename_without_extension)
+            elif header.value == "path_to_crop":
+                sheet.cell(row=next_row, column=i, value=path_to_crop)
+            elif header.value == "path_to_original":
+                if self.cfg['leafmachine']['use_RGB_label_images']:
+                    fname = os.path.basename(path_to_crop)
+                    base = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(path_to_crop))))
+                    path_to_original = os.path.join(base, 'Original_Images', fname)
+                    sheet.cell(row=next_row, column=i, value=path_to_original)
+                else:
+                    fname = os.path.basename(path_to_crop)
+                    base = os.path.dirname(os.path.dirname(path_to_crop))
+                    path_to_original = os.path.join(base, 'Original_Images', fname)
+                    sheet.cell(row=next_row, column=i, value=path_to_original)
+            elif header.value == "path_to_content":
+                sheet.cell(row=next_row, column=i, value=path_to_content)
+            elif header.value == "path_to_helper":
+                sheet.cell(row=next_row, column=i, value=path_to_helper)
+            elif header.value == "tokens_in":
+                sheet.cell(row=next_row, column=i, value=nt_in)
+            elif header.value == "tokens_out":
+                sheet.cell(row=next_row, column=i, value=nt_out)
+            elif header.value == "filename":
+                sheet.cell(row=next_row, column=i, value=filename_without_extension)
+
+            # "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_candidate_names","WFO_placement"
+            elif header.value in wfo_headers:
+                sheet.cell(row=next_row, column=i, value=WFO_record.get(header.value, ''))
+            # elif header.value == "WFO_exact_match":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_exact_match",''))
+            # elif header.value == "WFO_exact_match_name":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_exact_match_name",''))
+            # elif header.value == "WFO_best_match":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_best_match",''))
+            # elif header.value == "WFO_placement":
+            #     sheet.cell(row=next_row, column=i, value= WFO_record.get("WFO_placement",''))
+            elif header.value == "WFO_candidate_names":
+                candidate_names = WFO_record.get("WFO_candidate_names", '')
+                # Check if candidate_names is a list and convert to a string if it is
+                if isinstance(candidate_names, list):
+                    candidate_names_str = '|'.join(candidate_names)
+                else:
+                    candidate_names_str = candidate_names
+                sheet.cell(row=next_row, column=i, value=candidate_names_str)
+            
+            # "GEO_method", "GEO_formatted_full_string", "GEO_decimal_lat", "GEO_decimal_long",
+            # "GEO_city", "GEO_county", "GEO_state", "GEO_state_code", "GEO_country", "GEO_country_code", "GEO_continent"
+            elif header.value in geo_headers:
+                sheet.cell(row=next_row, column=i, value=GEO_record.get(header.value, ''))
+
+        # save the workbook
+        wb.save(path_transcription)
+    
+
+    def has_API_key(self, val):
+        if val != '':
+            return True
+        else:
+            return False
+
+
+    def set_API_keys(self):
+        self.dir_home = os.path.dirname(os.path.dirname(__file__))
+        self.path_cfg_private = os.path.join(self.dir_home, 'PRIVATE_DATA.yaml')
+        self.cfg_private = get_cfg_from_full_path(self.path_cfg_private)
+
+        self.has_key_openai = self.has_API_key(self.cfg_private['openai']['OPENAI_API_KEY'])
+
+        self.has_key_azure_openai = self.has_API_key(self.cfg_private['openai_azure']['api_version'])
+        
+        self.has_key_palm2 = self.has_API_key(self.cfg_private['google_palm']['google_palm_api'])
+
+        self.has_key_google_OCR = self.has_API_key(self.cfg_private['google_cloud']['path_json_file'])
+
+        self.has_key_mistral = self.has_API_key(self.cfg_private['mistral']['mistral_key'])
+
+        self.has_key_here = self.has_API_key(self.cfg_private['here']['api_key'])
+
+        self.has_open_cage_geocode = self.has_API_key(self.cfg_private['open_cage_geocode']['api_key'])
+
+        if self.has_key_openai:
+            openai.api_key = self.cfg_private['openai']['OPENAI_API_KEY']
+            os.environ["OPENAI_API_KEY"] = self.cfg_private['openai']['OPENAI_API_KEY']
+            
+        if self.has_key_azure_openai:
+            # os.environ["OPENAI_API_KEY"] = self.cfg_private['openai_azure']['openai_api_key']
+            self.llm = AzureChatOpenAI(
+                deployment_name = 'gpt-35-turbo',#'gpt-35-turbo',
+                openai_api_version = self.cfg_private['openai_azure']['api_version'],
+                openai_api_key = self.cfg_private['openai_azure']['openai_api_key'],
+                azure_endpoint = self.cfg_private['openai_azure']['openai_api_base'],
+                # openai_api_base=self.cfg_private['openai_azure']['openai_api_base'],
+                openai_organization = self.cfg_private['openai_azure']['openai_organization'],
+                # openai_api_type = self.cfg_private['openai_azure']['openai_api_type']
+            )
+        
+        # Enable this for all LLMs EXCEPT GOOGLE 
+        name_check = self.cfg['leafmachine']['LLM_version'].lower().split(' ')
+        if 'google' in name_check:
+            pass
+        else:
+            if self.has_key_google_OCR:
+                if os.path.exists(self.cfg_private['google_cloud']['path_json_file']):
+                    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.cfg_private['google_cloud']['path_json_file']
+                elif os.path.exists(self.cfg_private['google_cloud']['path_json_file_service_account2']):
+                    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = self.cfg_private['google_cloud']['path_json_file_service_account2']
+                else:
+                    raise f"Google JSON API key file not found"
+
+        if self.has_key_palm2:
+            os.environ['PALM'] = self.cfg_private['google_palm']['google_palm_api']
+            os.environ['PALM_PROJECT_ID'] = self.cfg_private['google_palm']['project_id']
+            os.environ['PALM_LOCATION'] = self.cfg_private['google_palm']['location']
+            os.environ['GOOGLE_API_KEY'] = self.cfg_private['google_palm']['google_palm_api']
+
+            # os.environ["GOOGLE_SERVICE_ACCOUNT"] = self.cfg_private['google_cloud']['path_json_file_service_account']
+            # vertexai.init(project='directed-curve-401601', location='us-central1')
+
+            # genai.configure(api_key=os.environ['PALM'])
+
+        if self.has_key_palm2:
+            os.environ['MISTRAL_API_KEY'] = self.cfg_private['mistral']['mistral_key']
+
+        if self.has_key_here:
+            os.environ['here_app_id'] = self.cfg_private['here']['app_id']
+            os.environ['here_api_key'] = self.cfg_private['here']['api_key']
+
+        if self.has_open_cage_geocode:
+            os.environ['open_cage_geocode'] = self.cfg_private['open_cage_geocode']['api_key']
+        
+
+        
+    # def initialize_embeddings(self):
+    #     '''Loading embedding search       __init__(self, db_name, path_domain_knowledge, logger, build_new_db=False, model_name="hkunlp/instructor-xl", device="cuda")'''
+    #     self.Voucher_Vision_Embedding = VoucherVisionEmbedding(self.db_name, self.path_domain_knowledge, logger=self.logger, build_new_db=self.build_new_db)
+
+
+    def clean_catalog_number(self, data, filename_without_extension):
+        #Cleans up the catalog number in data if it's a dict
+        
+        def modify_catalog_key(catalog_key, filename_without_extension, data):
+            # Helper function to apply modifications on catalog number
+            if catalog_key not in data:
+                new_data = {catalog_key: None}
+                data = {**new_data, **data}
+
+            if self.prefix_removal:
+                filename_without_extension = filename_without_extension.replace(self.prefix_removal, "")
+            if self.suffix_removal:
+                filename_without_extension = filename_without_extension.replace(self.suffix_removal, "")
+            if self.catalog_numerical_only:
+                filename_without_extension = self.remove_non_numbers(data[catalog_key])
+            data[catalog_key] = filename_without_extension
+            return data
+        
+        if isinstance(data, dict):
+            if self.headers_used == 'HEADERS_v1_n22':
+                return modify_catalog_key("Catalog Number", filename_without_extension, data)
+            elif self.headers_used in ['HEADERS_v2_n26', 'CUSTOM']:
+                return modify_catalog_key("filename", filename_without_extension, data)
+            else:
+                raise ValueError("Invalid headers used.")
+        else:
+            raise TypeError("Data is not of type dict.")
+        
+
+    def write_json_to_file(self, filepath, data):
+        '''Writes dictionary data to a JSON file.'''
+        with open(filepath, 'w') as txt_file:
+            if isinstance(data, dict):
+                data = json.dumps(data, indent=4, sort_keys=False)
+            txt_file.write(data)
+
+
+    # def create_null_json(self):
+    #     return {}
+    
+
+    def remove_non_numbers(self, s):
+        return ''.join([char for char in s if char.isdigit()])
+    
+
+    def create_null_row(self, filename_without_extension, path_to_crop, path_to_content, path_to_helper):
+        json_dict = {header: '' for header in self.headers} 
+        for header, value in json_dict.items():
+            if header == "path_to_crop":
+                json_dict[header] = path_to_crop
+            elif header == "path_to_original":
+                fname = os.path.basename(path_to_crop)
+                base = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(path_to_crop))))
+                path_to_original = os.path.join(base, 'Original_Images', fname)
+                json_dict[header] = path_to_original
+            elif header == "path_to_content":
+                json_dict[header] = path_to_content
+            elif header == "path_to_helper":
+                json_dict[header] = path_to_helper
+            elif header == "filename":
+                json_dict[header] = filename_without_extension
+
+            # "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_candidate_names","WFO_placement"
+            elif header == "WFO_exact_match":
+                json_dict[header] =''
+            elif header == "WFO_exact_match_name":
+                json_dict[header] = ''
+            elif header == "WFO_best_match":
+                json_dict[header] = ''
+            elif header == "WFO_candidate_names":
+                json_dict[header] = ''
+            elif header == "WFO_placement":
+                json_dict[header] = ''
+        return json_dict
+    
+
+    ##################################################################################################################################
+    ##################################################     OCR      ##################################################################
+    ##################################################################################################################################
+    def perform_OCR_and_save_results(self, image_index, jpg_file_path_OCR_helper, txt_file_path_OCR, txt_file_path_OCR_bounds):
+        self.logger.info(f'Working on {image_index + 1}/{len(self.img_paths)} --- Starting OCR')
+        # self.OCR - None
+
+        ### Process_image() runs the OCR for text, handwriting, trOCR AND creates the overlay image
+        ocr_google = OCRGoogle(self.path_to_crop, self.cfg, self.trOCR_model_version, self.trOCR_model, self.trOCR_processor, self.device)  
+        ocr_google.process_image(self.do_create_OCR_helper_image, self.logger)
+        self.OCR = ocr_google.OCR
+
+        self.write_json_to_file(txt_file_path_OCR, ocr_google.OCR_JSON_to_file)
+        
+        self.logger.info(f'Working on {image_index + 1}/{len(self.img_paths)} --- Finished OCR')
+
+        if len(self.OCR) > 0:
+            ocr_google.overlay_image.save(jpg_file_path_OCR_helper)
+
+            OCR_bounds = {}
+            if ocr_google.hand_text_to_box_mapping is not None:
+                OCR_bounds['OCR_bounds_handwritten'] = ocr_google.hand_text_to_box_mapping
+
+            if ocr_google.normal_text_to_box_mapping is not None:
+                OCR_bounds['OCR_bounds_printed'] = ocr_google.normal_text_to_box_mapping
+
+            if ocr_google.trOCR_text_to_box_mapping is not None:
+                OCR_bounds['OCR_bounds_trOCR'] = ocr_google.trOCR_text_to_box_mapping
+
+            self.write_json_to_file(txt_file_path_OCR_bounds, OCR_bounds)
+            self.logger.info(f'Working on {image_index + 1}/{len(self.img_paths)} --- Saved OCR Overlay Image')
+        else:
+            pass ########################################################################################################################### fix logic for no OCR
+
+    ##################################################################################################################################
+    #######################################################  LLM Switchboard  ########################################################
+    ##################################################################################################################################
+    def send_to_LLM(self, is_azure, progress_report, json_report, model_name):
+        self.n_failed_LLM_calls = 0
+        self.n_failed_OCR = 0
+
+        final_JSON_response = None
+        final_WFO_record = None
+        final_GEO_record = None
+
+        self.initialize_token_counters()
+        self.update_progress_report_initial(progress_report)
+
+        MODEL_NAME_FORMATTED = ModelMaps.get_API_name(model_name)
+        name_parts = model_name.split("_")
+        
+        self.setup_JSON_dict_structure()
+        
+        llm_model = self.initialize_llm_model(self.logger, MODEL_NAME_FORMATTED, self.JSON_dict_structure, name_parts, is_azure, self.llm)
+
+        prompts = []
+        for i, path_to_crop in enumerate(self.img_paths):
+            self.update_progress_report_batch(progress_report, i)
+
+            if self.should_skip_specimen(path_to_crop):
+                self.log_skipping_specimen(path_to_crop)
+                continue
+
+            paths = self.generate_paths(path_to_crop, i)
+            self.path_to_crop = path_to_crop
+
+            filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper = paths
+            self.perform_OCR_and_save_results(i, jpg_file_path_OCR_helper, txt_file_path_OCR, txt_file_path_OCR_bounds)
+
+            if not self.OCR:
+                self.n_failed_OCR += 1
+            else:
+                # Format prompt
+                prompt = self.setup_prompt()
+                prompt = remove_colons_and_double_apostrophes(prompt)
+                prompts.append(prompt)
+
+        # Process prompts in batch
+        if 'LOCAL' in name_parts and ('MISTRAL' in name_parts or 'MIXTRAL' in name_parts):
+            batch_results = llm_model.call_llm_local_MistralAI(prompts)  # Assuming this method is updated to handle batch processing
+
+        # Process each result from the batch
+        for i, result in enumerate(batch_results):
+            response_candidate, nt_in, nt_out, WFO_record, GEO_record = result.values()
+
+            self.n_failed_LLM_calls += 1 if response_candidate is None else 0
+
+            # Estimate n tokens returned
+            self.logger.info(f'Prompt tokens IN --- {nt_in}')
+            self.logger.info(f'Prompt tokens OUT --- {nt_out}')
+            
+            self.update_token_counters(nt_in, nt_out)
+
+            final_JSON_response, final_WFO_record, final_GEO_record = self.update_final_response(response_candidate, WFO_record, GEO_record, paths, path_to_crop, nt_in, nt_out)
+
+            self.log_completion_info(final_JSON_response)
+
+            json_report.set_JSON(final_JSON_response, final_WFO_record, final_GEO_record)
+
+        self.update_progress_report_final(progress_report)
+        final_JSON_response = self.parse_final_json_response(final_JSON_response)
+        return final_JSON_response, final_WFO_record, final_GEO_record, self.total_tokens_in, self.total_tokens_out
+
+    
+
+    ##################################################################################################################################
+    ################################################## LLM Helper Funcs ##############################################################
+    ##################################################################################################################################
+    def initialize_llm_model(self, logger, model_name, JSON_dict_structure, name_parts, is_azure=None, llm_object=None):
+        if 'LOCAL'in name_parts:
+            if ('MIXTRAL' in name_parts) or ('MISTRAL' in name_parts):
+                if 'CPU' in name_parts:
+                    return LocalCPUMistralHandler(logger, model_name, JSON_dict_structure)
+                else:
+                    return LocalMistralHandler(logger, model_name, JSON_dict_structure)
+        else:
+            if 'PALM2' in name_parts:
+                return GooglePalm2Handler(logger, model_name, JSON_dict_structure)
+            elif 'GEMINI' in name_parts:
+                return GoogleGeminiHandler(logger, model_name, JSON_dict_structure)
+            elif 'MISTRAL' in name_parts and ('LOCAL' not in name_parts):
+                return MistralHandler(logger, model_name, JSON_dict_structure)
+            else:
+                return OpenAIHandler(logger, model_name, JSON_dict_structure, is_azure, llm_object)
+
+    def setup_prompt(self):
+        Catalog = PromptCatalog()
+        prompt, _ = Catalog.prompt_SLTP(self.path_custom_prompts, OCR=self.OCR)
+        return prompt
+    
+    def setup_JSON_dict_structure(self):
+        Catalog = PromptCatalog()
+        _, self.JSON_dict_structure = Catalog.prompt_SLTP(self.path_custom_prompts, OCR='Text')
+    
+
+    def initialize_token_counters(self):
+        self.total_tokens_in = 0
+        self.total_tokens_out = 0
+
+
+    def update_progress_report_initial(self, progress_report):
+        if progress_report is not None:
+            progress_report.set_n_batches(len(self.img_paths))
+
+
+    def update_progress_report_batch(self, progress_report, batch_index):
+        if progress_report is not None:
+            progress_report.update_batch(f"Working on image {batch_index + 1} of {len(self.img_paths)}")
+
+
+    def should_skip_specimen(self, path_to_crop):
+        return os.path.basename(path_to_crop) in self.completed_specimens
+
+
+    def log_skipping_specimen(self, path_to_crop):
+        self.logger.info(f'[Skipping] specimen {os.path.basename(path_to_crop)} already processed')
+
+    
+    def update_token_counters(self, nt_in, nt_out):
+        self.total_tokens_in += nt_in
+        self.total_tokens_out += nt_out
+
+
+    def update_final_response(self, response_candidate, WFO_record, GEO_record, paths, path_to_crop, nt_in, nt_out):
+        filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper = paths
+        # Saving the JSON and XLSX files with the response and updating the final JSON response
+        if response_candidate is not None:
+            final_JSON_response_updated = self.save_json_and_xlsx(response_candidate, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
+            return final_JSON_response_updated, WFO_record, GEO_record
+        else:
+            final_JSON_response_updated = self.save_json_and_xlsx(response_candidate, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
+            return final_JSON_response_updated, WFO_record, GEO_record
+
+
+    def log_completion_info(self, final_JSON_response):
+        self.logger.info(f'Formatted JSON\n{final_JSON_response}')
+        self.logger.info(f'Finished API calls\n')
+
+
+    def update_progress_report_final(self, progress_report):
+        if progress_report is not None:
+            progress_report.reset_batch("Batch Complete")
+
+
+    def parse_final_json_response(self, final_JSON_response):
+        try:
+            return json.loads(final_JSON_response.strip('```').replace('json\n', '', 1).replace('json', '', 1))
+        except:
+            return final_JSON_response
+    
+    
+
+    def generate_paths(self, path_to_crop, i):
+        filename_without_extension = os.path.splitext(os.path.basename(path_to_crop))[0]
+        txt_file_path = os.path.join(self.Dirs.transcription_ind, filename_without_extension + '.json')
+        txt_file_path_OCR = os.path.join(self.Dirs.transcription_ind_OCR, filename_without_extension + '.json')
+        txt_file_path_OCR_bounds = os.path.join(self.Dirs.transcription_ind_OCR_bounds, filename_without_extension + '.json')
+        jpg_file_path_OCR_helper = os.path.join(self.Dirs.transcription_ind_OCR_helper, filename_without_extension + '.jpg')
+
+        self.logger.info(f'Working on {i+1}/{len(self.img_paths)} --- {filename_without_extension}')
+
+        return filename_without_extension, txt_file_path, txt_file_path_OCR, txt_file_path_OCR_bounds, jpg_file_path_OCR_helper
+
+
+    def save_json_and_xlsx(self, response, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out):
+        if response is None:
+            response = self.JSON_dict_structure
+            # Insert 'filename' as the first key
+            response = {'filename': filename_without_extension, **{k: v for k, v in response.items() if k != 'filename'}}
+            self.write_json_to_file(txt_file_path, response)
+
+            # Then add the null info to the spreadsheet
+            response_null = self.create_null_row(filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper)
+            self.add_data_to_excel_from_response(self.path_transcription, response_null, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in=0, nt_out=0)
+        
+        ### Set completed JSON
+        else:
+            response = self.clean_catalog_number(response, filename_without_extension)
+            self.write_json_to_file(txt_file_path, response)
+            # add to the xlsx file
+            self.add_data_to_excel_from_response(self.path_transcription, response, WFO_record, GEO_record, filename_without_extension, path_to_crop, txt_file_path, jpg_file_path_OCR_helper, nt_in, nt_out)
+        return response
+    
+
+    def process_specimen_batch(self, progress_report, json_report, is_real_run=False):
+        if not self.has_key:
+            self.logger.error(f'No API key found for {self.version_name}')
+            raise Exception(f"No API key found for {self.version_name}")
+
+        try:
+            if is_real_run:
+                progress_report.update_overall(f"Transcribing Labels")
+
+            final_json_response, final_WFO_record, final_GEO_record, total_tokens_in, total_tokens_out = self.send_to_LLM(self.is_azure, progress_report, json_report, self.model_name)
+            
+            return final_json_response, final_WFO_record, final_GEO_record, total_tokens_in, total_tokens_out
+
+        except Exception as e:
+            self.logger.error(f"LLM call failed in process_specimen_batch: {e}")
+            if progress_report is not None:
+                progress_report.reset_batch(f"Batch Failed")
+            self.close_logger_handlers()
+            raise
+
+
+    def close_logger_handlers(self):
+        for handler in self.logger.handlers[:]:
+            handler.close()
+            self.logger.removeHandler(handler)
+
+
+    def process_specimen_batch_OCR_test(self, path_to_crop):
+        for img_filename in os.listdir(path_to_crop):
+            img_path = os.path.join(path_to_crop, img_filename)
+        self.OCR, self.bounds, self.text_to_box_mapping = detect_text(img_path)
+
+
+
+def space_saver(cfg, Dirs, logger):
+    dir_out = cfg['leafmachine']['project']['dir_output']
+    run_name = Dirs.run_name
+
+    path_project = os.path.join(dir_out, run_name)
+
+    if cfg['leafmachine']['project']['delete_temps_keep_VVE']:
+        logger.name = '[DELETE TEMP FILES]'
+        logger.info("Deleting temporary files. Keeping files required for VoucherVisionEditor.")
+        delete_dirs = ['Archival_Components', 'Config_File']
+        for d in delete_dirs:
+            path_delete = os.path.join(path_project, d)
+            if os.path.exists(path_delete):
+                shutil.rmtree(path_delete)
+
+    elif cfg['leafmachine']['project']['delete_all_temps']:
+        logger.name = '[DELETE TEMP FILES]'
+        logger.info("Deleting ALL temporary files!")
+        delete_dirs = ['Archival_Components', 'Config_File', 'Original_Images', 'Cropped_Images']
+        for d in delete_dirs:
+            path_delete = os.path.join(path_project, d)
+            if os.path.exists(path_delete):
+                shutil.rmtree(path_delete)
+
+        # Delete the transctiption folder, but keep the xlsx
+        transcription_path = os.path.join(path_project, 'Transcription')
+        if os.path.exists(transcription_path):
+            for item in os.listdir(transcription_path):
+                item_path = os.path.join(transcription_path, item)
+                if os.path.isdir(item_path):  # if the item is a directory
+                    if os.path.exists(item_path):
+                        shutil.rmtree(item_path)  # delete the directory

vouchervision/utils_geolocate_HERE.py

@@ -0,0 +1,319 @@
+import os, requests
+import pycountry_convert as pc
+import unicodedata
+import pycountry_convert as pc
+import warnings
+
+
+def normalize_country_name(name):
+    return unicodedata.normalize('NFKD', name).encode('ASCII', 'ignore').decode('ASCII')
+
+def get_continent(country_name):
+    warnings.filterwarnings("ignore", category=UserWarning, module='pycountry')
+
+    continent_code_to_name = {
+        "AF": "Africa",
+        "NA": "North America",
+        "OC": "Oceania",
+        "AN": "Antarctica",
+        "AS": "Asia",
+        "EU": "Europe",
+        "SA": "South America"
+    }
+
+    try:
+        normalized_country_name = normalize_country_name(country_name)
+        # Get country alpha2 code
+        country_code = pc.country_name_to_country_alpha2(normalized_country_name)
+        # Get continent code from country alpha2 code
+        continent_code = pc.country_alpha2_to_continent_code(country_code)
+        # Map the continent code to continent name
+        return continent_code_to_name.get(continent_code, '')
+    except Exception as e:
+        print(str(e))
+        return ''
+    
+def validate_coordinates_here(record, replace_if_success_geo=False):
+    forward_url = 'https://geocode.search.hereapi.com/v1/geocode'
+    reverse_url = 'https://revgeocode.search.hereapi.com/v1/revgeocode'
+    
+    pinpoint = ['GEO_city','GEO_county','GEO_state','GEO_country',]
+    GEO_dict_null = {
+        'GEO_override_OCR': False,
+        'GEO_method': '',
+        'GEO_formatted_full_string': '',
+        'GEO_decimal_lat': '',
+        'GEO_decimal_long': '',
+        'GEO_city': '',
+        'GEO_county': '',
+        'GEO_state': '',
+        'GEO_state_code': '',
+        'GEO_country': '',
+        'GEO_country_code': '',
+        'GEO_continent': '',
+    }
+    GEO_dict = {
+        'GEO_override_OCR': False,
+        'GEO_method': '',
+        'GEO_formatted_full_string': '',
+        'GEO_decimal_lat': '',
+        'GEO_decimal_long': '',
+        'GEO_city': '',
+        'GEO_county': '',
+        'GEO_state': '',
+        'GEO_state_code': '',
+        'GEO_country': '',
+        'GEO_country_code': '',
+        'GEO_continent': '',
+    }
+    GEO_dict_rev = {
+        'GEO_override_OCR': False,
+        'GEO_method': '',
+        'GEO_formatted_full_string': '',
+        'GEO_decimal_lat': '',
+        'GEO_decimal_long': '',
+        'GEO_city': '',
+        'GEO_county': '',
+        'GEO_state': '',
+        'GEO_state_code': '',
+        'GEO_country': '',
+        'GEO_country_code': '',
+        'GEO_continent': '',
+    }
+    GEO_dict_rev_verbatim = {
+        'GEO_override_OCR': False,
+        'GEO_method': '',
+        'GEO_formatted_full_string': '',
+        'GEO_decimal_lat': '',
+        'GEO_decimal_long': '',
+        'GEO_city': '',
+        'GEO_county': '',
+        'GEO_state': '',
+        'GEO_state_code': '',
+        'GEO_country': '',
+        'GEO_country_code': '',
+        'GEO_continent': '',
+    }
+    GEO_dict_forward = {
+        'GEO_override_OCR': False,
+        'GEO_method': '',
+        'GEO_formatted_full_string': '',
+        'GEO_decimal_lat': '',
+        'GEO_decimal_long': '',
+        'GEO_city': '',
+        'GEO_county': '',
+        'GEO_state': '',
+        'GEO_state_code': '',
+        'GEO_country': '',
+        'GEO_country_code': '',
+        'GEO_continent': '',
+    }
+    GEO_dict_forward_locality = {
+        'GEO_override_OCR': False,
+        'GEO_method': '',
+        'GEO_formatted_full_string': '',
+        'GEO_decimal_lat': '',
+        'GEO_decimal_long': '',
+        'GEO_city': '',
+        'GEO_county': '',
+        'GEO_state': '',
+        'GEO_state_code': '',
+        'GEO_country': '',
+        'GEO_country_code': '',
+        'GEO_continent': '',
+    }
+
+    
+    # For production
+    query_forward = ', '.join(filter(None, [record.get('municipality', '').strip(),
+                                        record.get('county', '').strip(),
+                                        record.get('stateProvince', '').strip(),
+                                        record.get('country', '').strip()])).strip()
+    query_forward_locality = ', '.join(filter(None, [record.get('locality', '').strip(),
+                                        record.get('municipality', '').strip(),
+                                        record.get('county', '').strip(),
+                                        record.get('stateProvince', '').strip(),
+                                        record.get('country', '').strip()])).strip()
+    query_reverse = ','.join(filter(None, [record.get('decimalLatitude', '').strip(),
+                                        record.get('decimalLongitude', '').strip()])).strip()
+    query_reverse_verbatim = record.get('verbatimCoordinates', '').strip()
+    
+
+    '''
+    #For testing
+    # query_forward = 'Ann bor, michign'
+    query_forward = 'michigan'
+    query_forward_locality = 'Ann bor, michign'
+    # query_gps = "42 N,-83 W" # cannot have any spaces
+    # query_reverse_verbatim = "42.278366,-83.744718" # cannot have any spaces
+    query_reverse_verbatim = "42,-83" # cannot have any spaces
+    query_reverse = "42,-83" # cannot have any spaces
+    # params = {
+    #     'q': query_loc,
+    #     'apiKey': os.environ['here_api_key'],
+    # }'''
+
+    
+    params_rev = {
+        'at': query_reverse,
+        'apiKey': os.environ['here_api_key'],
+        'lang': 'en',
+    }
+    params_reverse_verbatim = {
+        'at': query_reverse_verbatim,
+        'apiKey': os.environ['here_api_key'],
+        'lang': 'en',
+    }
+    params_forward = {
+        'q': query_forward,
+        'apiKey': os.environ['here_api_key'],
+        'lang': 'en',
+    }
+    params_forward_locality = {
+        'q': query_forward_locality,
+        'apiKey': os.environ['here_api_key'],
+        'lang': 'en',
+    }
+
+    ### REVERSE
+    # If there are two string in the coordinates, try a reverse first based on the literal coordinates
+    response = requests.get(reverse_url, params=params_rev)
+    if response.status_code == 200:
+        data = response.json()
+        if data.get('items'):
+            first_result = data['items'][0]
+            GEO_dict_rev['GEO_method'] = 'HERE_Geocode_reverse'
+            GEO_dict_rev['GEO_formatted_full_string'] = first_result.get('title', '')
+            GEO_dict_rev['GEO_decimal_lat'] = first_result['position']['lat']
+            GEO_dict_rev['GEO_decimal_long'] = first_result['position']['lng']
+
+            address = first_result.get('address', {})
+            GEO_dict_rev['GEO_city'] = address.get('city', '')
+            GEO_dict_rev['GEO_county'] = address.get('county', '')
+            GEO_dict_rev['GEO_state'] = address.get('state', '')
+            GEO_dict_rev['GEO_state_code'] = address.get('stateCode', '')
+            GEO_dict_rev['GEO_country'] = address.get('countryName', '')
+            GEO_dict_rev['GEO_country_code'] = address.get('countryCode', '')
+            GEO_dict_rev['GEO_continent'] = get_continent(address.get('countryName', ''))
+
+    ### REVERSE Verbatim
+    # If there are two string in the coordinates, try a reverse first based on the literal coordinates
+    if GEO_dict_rev['GEO_city']: # If the reverse was successful, pass
+        GEO_dict = GEO_dict_rev
+    else:
+        response = requests.get(reverse_url, params=params_reverse_verbatim)
+        if response.status_code == 200:
+            data = response.json()
+            if data.get('items'):
+                first_result = data['items'][0]
+                GEO_dict_rev_verbatim['GEO_method'] = 'HERE_Geocode_reverse_verbatimCoordinates'
+                GEO_dict_rev_verbatim['GEO_formatted_full_string'] = first_result.get('title', '')
+                GEO_dict_rev_verbatim['GEO_decimal_lat'] = first_result['position']['lat']
+                GEO_dict_rev_verbatim['GEO_decimal_long'] = first_result['position']['lng']
+
+                address = first_result.get('address', {})
+                GEO_dict_rev_verbatim['GEO_city'] = address.get('city', '')
+                GEO_dict_rev_verbatim['GEO_county'] = address.get('county', '')
+                GEO_dict_rev_verbatim['GEO_state'] = address.get('state', '')
+                GEO_dict_rev_verbatim['GEO_state_code'] = address.get('stateCode', '')
+                GEO_dict_rev_verbatim['GEO_country'] = address.get('countryName', '')
+                GEO_dict_rev_verbatim['GEO_country_code'] = address.get('countryCode', '')
+                GEO_dict_rev_verbatim['GEO_continent'] = get_continent(address.get('countryName', ''))
+
+    ### FORWARD
+    ### Try forward, if failes, try reverse using deci, then verbatim
+    if GEO_dict_rev['GEO_city']: # If the reverse was successful, pass
+        GEO_dict = GEO_dict_rev
+    elif GEO_dict_rev_verbatim['GEO_city']:
+        GEO_dict = GEO_dict_rev_verbatim
+    else:
+        response = requests.get(forward_url, params=params_forward)
+        if response.status_code == 200:
+            data = response.json()
+            if data.get('items'):
+                first_result = data['items'][0]
+                GEO_dict_forward['GEO_method'] = 'HERE_Geocode_forward'
+                GEO_dict_forward['GEO_formatted_full_string'] = first_result.get('title', '')
+                GEO_dict_forward['GEO_decimal_lat'] = first_result['position']['lat']
+                GEO_dict_forward['GEO_decimal_long'] = first_result['position']['lng']
+
+                address = first_result.get('address', {})
+                GEO_dict_forward['GEO_city'] = address.get('city', '')
+                GEO_dict_forward['GEO_county'] = address.get('county', '')
+                GEO_dict_forward['GEO_state'] = address.get('state', '')
+                GEO_dict_forward['GEO_state_code'] = address.get('stateCode', '')
+                GEO_dict_forward['GEO_country'] = address.get('countryName', '')
+                GEO_dict_forward['GEO_country_code'] = address.get('countryCode', '')
+                GEO_dict_forward['GEO_continent'] = get_continent(address.get('countryName', ''))
+
+    ### FORWARD locality
+    ### Try forward, if failes, try reverse using deci, then verbatim
+    if GEO_dict_rev['GEO_city']: # If the reverse was successful, pass
+        GEO_dict = GEO_dict_rev
+    elif GEO_dict_rev_verbatim['GEO_city']:
+        GEO_dict = GEO_dict_rev_verbatim
+    elif GEO_dict_forward['GEO_city']:
+        GEO_dict = GEO_dict_forward
+    else:
+        response = requests.get(forward_url, params=params_forward_locality)
+        if response.status_code == 200:
+            data = response.json()
+            if data.get('items'):
+                first_result = data['items'][0]
+                GEO_dict_forward_locality['GEO_method'] = 'HERE_Geocode_forward_locality'
+                GEO_dict_forward_locality['GEO_formatted_full_string'] = first_result.get('title', '')
+                GEO_dict_forward_locality['GEO_decimal_lat'] = first_result['position']['lat']
+                GEO_dict_forward_locality['GEO_decimal_long'] = first_result['position']['lng']
+
+                address = first_result.get('address', {})
+                GEO_dict_forward_locality['GEO_city'] = address.get('city', '')
+                GEO_dict_forward_locality['GEO_county'] = address.get('county', '')
+                GEO_dict_forward_locality['GEO_state'] = address.get('state', '')
+                GEO_dict_forward_locality['GEO_state_code'] = address.get('stateCode', '')
+                GEO_dict_forward_locality['GEO_country'] = address.get('countryName', '')
+                GEO_dict_forward_locality['GEO_country_code'] = address.get('countryCode', '')
+                GEO_dict_forward_locality['GEO_continent'] = get_continent(address.get('countryName', ''))
+
+        
+    # print(json.dumps(GEO_dict,indent=4))
+    
+
+    # Pick the most detailed version 
+    # if GEO_dict_rev['GEO_formatted_full_string'] and GEO_dict_forward['GEO_formatted_full_string']:
+    for loc in pinpoint:
+        rev = GEO_dict_rev.get(loc,'')
+        forward = GEO_dict_forward.get(loc,'')
+        forward_locality = GEO_dict_forward_locality.get(loc,'')
+        rev_verbatim = GEO_dict_rev_verbatim.get(loc,'')
+
+        if not rev and not forward and not forward_locality and not rev_verbatim:
+            pass
+        elif rev:
+            GEO_dict = GEO_dict_rev
+            break
+        elif forward:
+            GEO_dict = GEO_dict_forward
+            break
+        elif forward_locality:
+            GEO_dict = GEO_dict_forward_locality
+            break
+        elif rev_verbatim:
+            GEO_dict = GEO_dict_rev_verbatim
+            break
+        else:
+            GEO_dict = GEO_dict_null
+            
+
+    if GEO_dict['GEO_formatted_full_string'] and replace_if_success_geo:
+        GEO_dict['GEO_override_OCR'] = True
+        record['country'] = GEO_dict.get('GEO_country')
+        record['stateProvince'] = GEO_dict.get('GEO_state')
+        record['county'] = GEO_dict.get('GEO_county')
+        record['municipality'] = GEO_dict.get('GEO_city')
+
+    # print(json.dumps(GEO_dict,indent=4))
+    return record, GEO_dict
+
+
+if __name__ == "__main__":
+    validate_coordinates_here(None)

vouchervision/utils_geolocate_OpenCage.py

@@ -0,0 +1,95 @@
+import os
+from opencage.geocoder import OpenCageGeocode
+import pycountry_convert as pc
+import warnings
+import unicodedata
+import pycountry_convert as pc
+import warnings
+
+
+### TODO 1/24/24
+### If I want to use this instead of HERE, update the procedure for picking the best/most granular geolocation
+
+
+def normalize_country_name(name):
+    return unicodedata.normalize('NFKD', name).encode('ASCII', 'ignore').decode('ASCII')
+
+def get_continent(country_name):
+    warnings.filterwarnings("ignore", category=UserWarning, module='pycountry')
+
+    continent_code_to_name = {
+        "AF": "Africa",
+        "NA": "North America",
+        "OC": "Oceania",
+        "AN": "Antarctica",
+        "AS": "Asia",
+        "EU": "Europe",
+        "SA": "South America"
+    }
+
+    try:
+        normalized_country_name = normalize_country_name(country_name)
+        # Get country alpha2 code
+        country_code = pc.country_name_to_country_alpha2(normalized_country_name)
+        # Get continent code from country alpha2 code
+        continent_code = pc.country_alpha2_to_continent_code(country_code)
+        # Map the continent code to continent name
+        return continent_code_to_name.get(continent_code, '')
+    except Exception as e:
+        print(str(e))
+        return ''
+    
+def validate_coordinates_opencage(record, replace_if_success_geo=False):
+    GEO_dict = {
+        'GEO_method': '',
+        'GEO_formatted_full_string': '',
+        'GEO_decimal_lat': '',
+        'GEO_decimal_long': '',
+        'GEO_city': '',
+        'GEO_county': '',
+        'GEO_state': '',
+        'GEO_state_code': '',
+        'GEO_country': '',
+        'GEO_country_code': '',
+        'GEO_continent': '',
+    }
+    
+    geocoder = OpenCageGeocode(os.environ['open_cage_geocode'])
+
+    query_loc = ', '.join(filter(None, [record.get('municipality', '').strip(), 
+                                        record.get('county', '').strip(), 
+                                        record.get('stateProvince', '').strip(), 
+                                        record.get('country', '').strip()])).strip()
+    
+    
+    query_decimal = ', '.join(filter(None, [record.get('decimalLatitude', '').strip(), 
+                                        record.get('decimalLongitude', '').strip()])).strip()
+    query_verbatim = record.get('verbatimCoordinates', '').strip()
+
+    # results = geocoder.geocode('Ann Arbor, Michigan', no_annotations='1')
+    results = geocoder.geocode(query_loc, no_annotations='1')
+
+    if results:
+        GEO_dict['GEO_method'] = 'OpenCageGeocode_forward'
+        GEO_dict['GEO_formatted_full_string'] = results[0]['formatted']
+        GEO_dict['GEO_decimal_lat'] = results[0]['geometry']['lat']
+        GEO_dict['GEO_decimal_long'] = results[0]['geometry']['lng']
+
+        GEO_dict['GEO_city'] = results[0]['components']['city']
+        GEO_dict['GEO_county'] = results[0]['components']['county']
+        GEO_dict['GEO_state'] = results[0]['components']['state']
+        GEO_dict['GEO_state_code'] = results[0]['components']['state_code']
+        GEO_dict['GEO_country'] = results[0]['components']['country']
+        GEO_dict['GEO_country_code'] = results[0]['components']['country_code']
+        GEO_dict['GEO_continent'] = results[0]['components']['continent']
+    
+    if GEO_dict['GEO_formatted_full_string'] and replace_if_success_geo:
+        GEO_dict['GEO_override_OCR'] = True
+        record['country'] = GEO_dict.get('GEO_country')
+        record['stateProvince'] = GEO_dict.get('GEO_state')
+        record['county'] = GEO_dict.get('GEO_county')
+        record['municipality'] = GEO_dict.get('GEO_city')
+
+    return record, GEO_dict
+
+

vouchervision/utils_taxonomy_WFO.py

@@ -0,0 +1,319 @@
+import requests
+from urllib.parse import urlencode
+from Levenshtein import ratio
+from fuzzywuzzy import fuzz
+
+class WFONameMatcher:
+    def __init__(self):
+        self.base_url = "https://list.worldfloraonline.org/matching_rest.php?"
+        self.N_BEST_CANDIDATES = 10
+        self.NULL_DICT = {
+                        "WFO_exact_match": False,
+                        "WFO_exact_match_name": "",
+                        "WFO_candidate_names": "",
+                        "WFO_best_match": "",
+                        "WFO_placement": "",
+                        "WFO_override_OCR": False,
+                    }
+        self.SEP = '|'
+
+    def extract_input_string(self, record):
+        primary_input = f"{record.get('scientificName', '').strip()} {record.get('scientificNameAuthorship', '').strip()}".strip()
+        secondary_input = ' '.join(filter(None, [record.get('genus', '').strip(), 
+                                                 record.get('subgenus', '').strip(), 
+                                                 record.get('specificEpithet', '').strip(), 
+                                                 record.get('infraspecificEpithet', '').strip()])).strip()
+
+        return primary_input, secondary_input
+
+    def query_wfo_name_matching(self, input_string, check_homonyms=True, check_rank=True, accept_single_candidate=True):
+        params = {
+            "input_string": input_string,
+            "check_homonyms": check_homonyms,
+            "check_rank": check_rank,
+            "method": "full",
+            "accept_single_candidate": accept_single_candidate,
+        }
+
+        full_url = self.base_url + urlencode(params)
+
+        response = requests.get(full_url)
+        if response.status_code == 200:
+            return response.json()
+        else:
+            return {"error": True, "message": "Failed to fetch data from WFO API"}
+    
+    def query_and_process(self, record):
+        primary_input, secondary_input = self.extract_input_string(record)
+        
+        # Query with primary input
+        primary_result = self.query_wfo_name_matching(primary_input)
+        primary_processed, primary_ranked_candidates = self.process_wfo_response(primary_result, primary_input)
+
+        if primary_processed.get('WFO_exact_match'):
+            print("Selected Primary --- Exact Primary & Unchecked Secondary")
+            return primary_processed
+        else:
+            # Query with secondary input
+            secondary_result = self.query_wfo_name_matching(secondary_input)
+            secondary_processed, secondary_ranked_candidates = self.process_wfo_response(secondary_result, secondary_input)
+
+            if secondary_processed.get('WFO_exact_match'):
+                print("Selected Secondary --- Unchecked Primary & Exact Secondary")
+                return secondary_processed
+            
+            else:
+                # Both failed, just return the first failure
+                if (primary_processed.get("WFO_candidate_names") == '') and (secondary_processed.get("WFO_candidate_names") == ''):
+                    print("Selected Primary --- Failed Primary & Failed Secondary")
+                    return primary_processed
+                
+                # 1st failed, just return the second
+                elif (primary_processed.get("WFO_candidate_names") == '') and (len(secondary_processed.get("WFO_candidate_names")) > 0):
+                    print("Selected Secondary --- Failed Primary & Partial Secondary")
+                    return secondary_processed
+                
+                # 2nd failed, just return the first
+                elif (len(primary_processed.get("WFO_candidate_names")) > 0) and (secondary_processed.get("WFO_candidate_names") == ''):
+                    print("Selected Primary --- Partial Primary & Failed Secondary")
+                    return primary_processed
+
+                # Both have partial matches, compare and rerank
+                elif (len(primary_processed.get("WFO_candidate_names")) > 0) and (len(secondary_processed.get("WFO_candidate_names")) > 0):
+                    # Combine and sort results, ensuring no duplicates
+                    combined_candidates = list(set(primary_ranked_candidates + secondary_ranked_candidates))
+                    combined_candidates.sort(key=lambda x: (x[1], x[0]), reverse=True)  # Sort by similarity score, then name
+                    
+                    # Replace candidates with combined_candidates and combined best match 
+                    best_score_primary = primary_processed["WFO_candidate_names"][0][1]
+                    best_score_secondary = secondary_processed["WFO_candidate_names"][0][1]
+
+                    # Extracting only the candidate names from the top candidates
+                    top_candidates = combined_candidates[:self.N_BEST_CANDIDATES]
+                    cleaned_candidates = [cand[0] for cand in top_candidates]
+
+                    if best_score_primary >= best_score_secondary:
+                            
+                        primary_processed["WFO_candidate_names"] = cleaned_candidates
+                        primary_processed["WFO_best_match"] = cleaned_candidates[0]
+                        
+                        response_placement = self.query_wfo_name_matching(primary_processed["WFO_best_match"])
+                        placement_exact_match = response_placement.get("match")
+                        primary_processed["WFO_placement"] = placement_exact_match.get("placement", '')
+                        
+                        print("Selected Primary --- Partial Primary & Partial Secondary")
+                        return primary_processed
+                    else:
+                        secondary_processed["WFO_candidate_names"] = cleaned_candidates
+                        secondary_processed["WFO_best_match"] = cleaned_candidates[0]
+
+                        response_placement = self.query_wfo_name_matching(secondary_processed["WFO_best_match"])
+                        placement_exact_match = response_placement.get("match")
+                        secondary_processed["WFO_placement"] = placement_exact_match.get("placement", '')
+
+                        print("Selected Secondary --- Partial Primary & Partial Secondary")
+                        return secondary_processed
+                else:
+                    return self.NULL_DICT
+
+    def process_wfo_response(self, response, query):
+        simplified_response = {}
+        ranked_candidates = None
+
+        exact_match = response.get("match")
+        simplified_response["WFO_exact_match"] = bool(exact_match)
+
+        candidates = response.get("candidates", [])
+        candidate_names = [candidate["full_name_plain"] for candidate in candidates] if candidates else []
+
+        if not exact_match and candidate_names:
+            cleaned_candidates, ranked_candidates = self._rank_candidates_by_similarity(query, candidate_names)
+            simplified_response["WFO_candidate_names"] = cleaned_candidates
+            simplified_response["WFO_best_match"] = cleaned_candidates[0] if cleaned_candidates else ''
+        elif exact_match:
+            simplified_response["WFO_candidate_names"] = exact_match.get("full_name_plain")
+            simplified_response["WFO_best_match"] = exact_match.get("full_name_plain")
+        else:
+            simplified_response["WFO_candidate_names"] = ''
+            simplified_response["WFO_best_match"] = ''
+
+        # Call WFO again to update placement using WFO_best_match
+        try:
+            response_placement = self.query_wfo_name_matching(simplified_response["WFO_best_match"])
+            placement_exact_match = response_placement.get("match")
+            simplified_response["WFO_placement"] = placement_exact_match.get("placement", '')
+        except:
+            simplified_response["WFO_placement"] = ''
+
+        return simplified_response, ranked_candidates
+    
+    def _rank_candidates_by_similarity(self, query, candidates):
+        string_similarities = []
+        fuzzy_similarities = {candidate: fuzz.ratio(query, candidate) for candidate in candidates}
+        query_words = query.split()
+
+        for candidate in candidates:
+            candidate_words = candidate.split()
+            # Calculate word similarities and sum them up
+            word_similarities = [ratio(query_word, candidate_word) for query_word, candidate_word in zip(query_words, candidate_words)]
+            total_word_similarity = sum(word_similarities)
+
+            # Calculate combined similarity score (average of word and fuzzy similarities)
+            fuzzy_similarity = fuzzy_similarities[candidate]
+            combined_similarity = (total_word_similarity + fuzzy_similarity) / 2
+            string_similarities.append((candidate, combined_similarity))
+
+        # Sort the candidates based on combined similarity, higher scores first
+        ranked_candidates = sorted(string_similarities, key=lambda x: x[1], reverse=True)
+
+        # Extracting only the candidate names from the top candidates
+        top_candidates = ranked_candidates[:self.N_BEST_CANDIDATES]
+        cleaned_candidates = [cand[0] for cand in top_candidates]
+        
+        return cleaned_candidates, ranked_candidates
+    
+    def check_WFO(self, record, replace_if_success_wfo):
+        self.replace_if_success_wfo = replace_if_success_wfo
+
+        # "WFO_exact_match","WFO_exact_match_name","WFO_best_match","WFO_candidate_names","WFO_placement"
+        simplified_response = self.query_and_process(record)
+        simplified_response['WFO_override_OCR'] = False
+
+        # best_match
+        if simplified_response.get('WFO_exact_match'):
+            simplified_response['WFO_exact_match_name'] = simplified_response.get('WFO_best_match')
+        else:
+            simplified_response['WFO_exact_match_name'] = ''
+
+        # placement
+        wfo_placement = simplified_response.get('WFO_placement', '')
+        if wfo_placement:
+            parts = wfo_placement.split('/')[1:]
+            simplified_response['WFO_placement'] = self.SEP.join(parts)
+        else:
+            simplified_response['WFO_placement'] = ''
+
+        if simplified_response.get('WFO_exact_match') and replace_if_success_wfo:
+            simplified_response['WFO_override_OCR'] = True
+            name_parts = simplified_response.get('WFO_placement').split('$')[0]
+            name_parts = name_parts.split(self.SEP)
+            record['order'] = name_parts[3]
+            record['family'] = name_parts[4]
+            record['genus'] = name_parts[5]
+            record['specificEpithet'] = name_parts[6]
+            record['scientificName'] = simplified_response.get('WFO_exact_match_name')
+
+        return record, simplified_response
+    
+def validate_taxonomy_WFO(record_dict, replace_if_success_wfo=False):
+    Matcher = WFONameMatcher()
+    try:    
+        record_dict, WFO_dict = Matcher.check_WFO(record_dict, replace_if_success_wfo)
+        return record_dict, WFO_dict
+    except:
+        return record_dict, Matcher.NULL_DICT
+
+'''
+if __name__ == "__main__":
+    Matcher = WFONameMatcher()
+    # input_string = "Rhopalocarpus alterfolius"
+    record_exact_match ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "Hypericum prolificum",
+        "scientificNameAuthorship": "",
+
+        "genus": "Hypericum",
+        "subgenus": "",
+        "specificEpithet": "prolificum",
+        "infraspecificEpithet": "",
+    }
+    record_partialPrimary_exactSecondary ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "Hyperic prolificum",
+        "scientificNameAuthorship": "",
+
+        "genus": "Hypericum",
+        "subgenus": "",
+        "specificEpithet": "prolificum",
+        "infraspecificEpithet": "",
+    }
+    record_exactPrimary_partialSecondary ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "Hypericum prolificum",
+        "scientificNameAuthorship": "",
+
+        "genus": "Hyperic",
+        "subgenus": "",
+        "specificEpithet": "prolificum",
+        "infraspecificEpithet": "",
+    }
+    record_partialPrimary_partialSecondary ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "Hyperic prolificum",
+        "scientificNameAuthorship": "",
+
+        "genus": "Hypericum",
+        "subgenus": "",
+        "specificEpithet": "prolific",
+        "infraspecificEpithet": "",
+    }
+    record_partialPrimary_partialSecondary_swap ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "Hypericum prolific",
+        "scientificNameAuthorship": "",
+
+        "genus": "Hyperic",
+        "subgenus": "",
+        "specificEpithet": "prolificum",
+        "infraspecificEpithet": "",
+    }
+    record_errorPrimary_partialSecondary ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "ricum proli",
+        "scientificNameAuthorship": "",
+
+        "genus": "Hyperic",
+        "subgenus": "",
+        "specificEpithet": "prolificum",
+        "infraspecificEpithet": "",
+    }
+    record_partialPrimary_errorSecondary ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "Hyperic prolificum",
+        "scientificNameAuthorship": "",
+
+        "genus": "ricum",
+        "subgenus": "",
+        "specificEpithet": "proli",
+        "infraspecificEpithet": "",
+    }
+    record_errorPrimary_errorSecondary ={
+        "order": "Malpighiales",
+        "family": "Hypericaceae",
+        "scientificName": "ricum proli",
+        "scientificNameAuthorship": "",
+
+        "genus": "ricum",
+        "subgenus": "",
+        "specificEpithet": "proli",
+        "infraspecificEpithet": "",
+    }
+    options = [record_exact_match,
+               record_partialPrimary_exactSecondary,
+               record_exactPrimary_partialSecondary,
+               record_partialPrimary_partialSecondary,
+               record_partialPrimary_partialSecondary_swap,
+               record_errorPrimary_partialSecondary,
+               record_partialPrimary_errorSecondary,
+               record_errorPrimary_errorSecondary]
+    for opt in options:
+        simplified_response = Matcher.check_WFO(opt)
+        print(json.dumps(simplified_response, indent=4))
+'''

vouchervision/vouchervision_main.py

@@ -8,7 +8,7 @@ parentdir = os.path.dirname(currentdir)
 sys.path.append(parentdir)
 sys.path.append(currentdir)
 from vouchervision.component_detector.component_detector import detect_plant_components, detect_archival_components
-from general_utils import make_zipfile, add_to_expense_report, save_token_info_as_csv, print_main_start, check_for_subdirs_VV, load_config_file, load_config_file_testing, report_config, save_config_file, subset_dir_images, crop_detections_from_images_VV
+from general_utils import add_to_expense_report, save_token_info_as_csv, print_main_start, check_for_subdirs_VV, load_config_file, load_config_file_testing, report_config, save_config_file, subset_dir_images, crop_detections_from_images_VV
 from directory_structure_VV import Dir_Structure
 from data_project import Project_Info
 from LM2_logger import start_logging
@@ -16,10 +16,7 @@ from fetch_data import fetch_data
 from utils_VoucherVision import VoucherVision, space_saver
 
 
-def voucher_vision(cfg_file_path, dir_home, path_custom_prompts, cfg_test, progress_report, path_api_cost=None, test_ind = None, is_real_run=False):
-    # get_n_overall = progress_report.get_n_overall()
-    # progress_report.update_overall(f"Working on {test_ind+1} of {get_n_overall}")
-
+def voucher_vision(cfg_file_path, dir_home, path_custom_prompts, cfg_test, progress_report, json_report, path_api_cost=None, test_ind = None, is_hf = True, is_real_run=False):
     t_overall = perf_counter()
 
     # Load config file
@@ -29,19 +26,12 @@ def voucher_vision(cfg_file_path, dir_home, path_custom_prompts, cfg_test, progr
         cfg = load_config_file(dir_home, cfg_file_path, system='VoucherVision')  # For VoucherVision
     else:
         cfg = cfg_test 
-    # user_cfg = load_config_file(dir_home, cfg_file_path)
-    # cfg = Config(user_cfg)
 
     # Check to see if there are subdirs
     # Yes --> use the names of the subsirs as run_name
     run_name, dirs_list, has_subdirs = check_for_subdirs_VV(cfg)
     print(f"run_name {run_name} dirs_list{dirs_list} has_subdirs{has_subdirs}")
 
-    # for dir_ind, dir_in in enumerate(dirs_list):
-    #     if has_subdirs:
-    #         cfg['leafmachine']['project']['dir_images_local'] = dir_in
-    #         cfg['leafmachine']['project']['run_name'] = run_name[dir_ind]
-
     # Dir structure
     if is_real_run:
         progress_report.update_overall(f"Creating Output Directory Structure")
@@ -67,21 +57,16 @@ def voucher_vision(cfg_file_path, dir_home, path_custom_prompts, cfg_test, progr
     # Detect Archival Components
     print_main_start("Locating Archival Components")
     Project = detect_archival_components(cfg, logger, dir_home, Project, Dirs, is_real_run, progress_report)
- 
+
     # Save cropped detections
     crop_detections_from_images_VV(cfg, logger, dir_home, Project, Dirs)
 
     # Process labels
-    Voucher_Vision = VoucherVision(cfg, logger, dir_home, path_custom_prompts, Project, Dirs)
+    Voucher_Vision = VoucherVision(cfg, logger, dir_home, path_custom_prompts, Project, Dirs, is_hf)
     n_images = len(Voucher_Vision.img_paths)
-    last_JSON_response, total_tokens_in, total_tokens_out = Voucher_Vision.process_specimen_batch(progress_report, is_real_run)
-    
-    if path_api_cost:
-        cost_summary, data, total_cost = save_token_info_as_csv(Dirs, cfg['leafmachine']['LLM_version'], path_api_cost, total_tokens_in, total_tokens_out, n_images)
-        add_to_expense_report(dir_home, data)
-        logger.info(cost_summary)
-    else:
-        total_cost = None #TODO add config tests to expense_report
+    last_JSON_response, final_WFO_record, final_GEO_record, total_tokens_in, total_tokens_out = Voucher_Vision.process_specimen_batch(progress_report, json_report, is_real_run)
+
+    total_cost = save_token_info_as_csv(Dirs, cfg['leafmachine']['LLM_version'], path_api_cost, total_tokens_in, total_tokens_out, n_images, dir_home, logger)
 
     t_overall_s = perf_counter()
     logger.name = 'Run Complete! :)'
@@ -89,20 +74,11 @@ def voucher_vision(cfg_file_path, dir_home, path_custom_prompts, cfg_test, progr
     space_saver(cfg, Dirs, logger)
 
     if is_real_run:
-        progress_report.update_overall(f"Run Complete! :)")
-
-    for handler in logger.handlers[:]:
-        handler.close()
-        logger.removeHandler(handler)
+        progress_report.update_overall(f"Run Complete!")
 
-    # Create Higging Face zip file
-    dir_to_zip = os.path.join(Dirs.dir_home, Dirs.run_name)  
-    zip_filename = Dirs.run_name  
+    Voucher_Vision.close_logger_handlers()
 
-    # Creating a zip file
-    zip_filepath = make_zipfile(dir_to_zip, zip_filename)
-
-    return last_JSON_response, total_cost, zip_filepath
+    return last_JSON_response, final_WFO_record, final_GEO_record, total_cost, Voucher_Vision.n_failed_OCR, Voucher_Vision.n_failed_LLM_calls 
 
 def voucher_vision_OCR_test(cfg_file_path, dir_home, cfg_test, path_to_crop):
     # get_n_overall = progress_report.get_n_overall()
@@ -157,7 +133,6 @@ def voucher_vision_OCR_test(cfg_file_path, dir_home, cfg_test, path_to_crop):
     Voucher_Vision = VoucherVision(cfg, logger, dir_home, None, Project, Dirs)
     last_JSON_response = Voucher_Vision.process_specimen_batch_OCR_test(path_to_crop)
 
-
 if __name__ == '__main__':    
     is_test = False