Create README.md

README.md

@@ -0,0 +1,111 @@
+---
+license: mit
+tags:
+- medical
+---
+# MedImageInsight: Open-Source Medical Image Embedding Model
+
+This repository provides a simplified implementation for using the MedImageInsight model, an open-source medical imaging embedding model presented in the paper [MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging](https://arxiv.org/abs/2410.06542) by Noel C. F. Codella et al. The official guide to access the model from Microsoft is quite complicated, and it is arguable whether the model is truly open-source. This repository aims to make it easier to use the MedImageInsight model for various tasks, such as zero-shot classification, image embedding, and text embedding.
+
+What we have done:
+
+- Downloaded the models from azure
+- Got rid of all the uncecessary files
+- Got rid of uncecessary mlflow code to make standalone implementation
+- Moved to uv for dependecy management
+- Added multi-label classification
+- Create example FastAPI service
+
+## Usage
+
+1. Clone the repository and navigate to the project directory.
+
+Make sure you have git-lfs installed (https://git-lfs.com)
+```bash
+git lfs install
+```
+
+```bash
+git clone https://huggingface.co/lion-ai/MedImageInsights
+```
+3. Install the required dependencies
+We are using [uv](https://github.com/astral-sh/uv) package manager to simplyfy the installation.
+
+To create a virual env simply run
+```bash
+uv sync
+```
+Or to run a single script just run
+```bash
+uv run example.py
+```
+
+Thats it!
+
+## Examples
+See to the `example.py` file.
+### Zero-shot image classification
+
+Here's an example of how to use the `MedImageInsight` class for zero-shot classification:
+
+```python
+# Initialize classifier
+classifier = MedImageInsight(
+    model_dir="2024.09.27",
+    vision_model_name="medimageinsigt-v1.0.0.pt",
+    language_model_name="language_model.pth"
+)
+
+# Load model
+classifier.load_model()
+
+# Read image
+image = base64.encodebytes(read_image("image.png")).decode("utf-8")
+
+# Zero-shot classification
+images = [image]
+labels = ["normal", "Pneumonia", "unclear"]
+results = classifier.predict(images, labels)
+print(results)
+```
+###  Multi-label zero-shot image classification
+Run multi-label image classification (without softmax at the end)
+```python
+# Multilabel classification example
+images = [image]
+labels = ["normal", "Pneumonia", "Fracture", "Tumor"]
+results = classifier.predict(images, labels, multilabel=True)
+print(results)
+
+```
+### Image embeddings
+
+```python
+results = classifier.encode(images=images)
+print(results["image_embeddings"])
+```
+
+### Text embeddings
+```python
+results = classifier.encode(texts=labels)
+print(results["text_embeddings"])
+```
+### Flask FastAPI server
+```bash
+uv run flask_app.py
+```
+Go to to localhost:8000/docs to see the swagger.
+
+The application provides endpoints for classification and image embeddings. Images have to be base64 encoded
+
+
+## Roadmap
+- [x] Basic implementation
+- [x] Multilabel classification
+- [x] FastAPI service
+- [ ] HF compatible API (from_pretrained())
+- [ ] Explainability
+
+## Acknowledgments
+
+This repository is based on the work presented in the paper "MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging" by Noel C. F. Codella et al. ([arXiv:2410.06542](https://arxiv.org/abs/2410.06542)).