README.md

---
license: other
---

# Overview
This project aims to support visually impaired individuals in their daily navigation. 

This project combines the [YOLO](https://ultralytics.com/yolov8) model and [LLaMa 2 7b](https://huggingface.co/meta-llama/Llama-2-7b) for the navigation.

YOLO is trained on the bounding box data from the [AI Hub](https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=189),
Output of YOLO (bbox data) is converted as lists like `[[class_of_obj_1, xmin, xmax, ymin, ymax, size], [class_of...] ...]` then added to the input of question.
The LLM is trained to navigate using [LearnItAnyway/Visual-Navigation-21k](https://huggingface.co/datasets/LearnItAnyway/Visual-Navigation-21k) multi-turn dataset


## Usage
We show how to use the model in [yolo_llama_visnav_test.ipynb](https://huggingface.co/LearnItAnyway/YOLO_LLaMa_7B_VisNav/blob/main/yolo_llama_visnav_test.ipynb)