File size: 2,741 Bytes
64b04d1 57b9817 3b007f8 84d5c1c 64b04d1 10d3188 64b04d1 cc33b4e 64b04d1 10d3188 64b04d1 72be1d8 5332b77 3ed8821 9991254 64b04d1 e8a7d1f 64b04d1 cc33b4e 64b04d1 cc33b4e 64b04d1 cc33b4e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
license: apache-2.0
datasets:
- remyxai/vqasynth_spacellava
tags:
- remyx
base_model:
- liuhaotian/llava-v1.5-13b
---
# Model Card for SpaceLLaVA
**SpaceLLaVA** uses LoRA to fine-tune [LLaVA](https://github.com/haotian-liu/LLaVA/tree/main) on a [dataset](https://huggingface.co/datasets/remyxai/vqasynth_spacellava) designed with [VQASynth](https://github.com/remyxai/VQASynth/tree/main) to enhance spatial reasoning as in [SpatialVLM](https://spatial-vlm.github.io/)
## Model Details
### Model Description
This model uses data synthesis techniques and publically available models to reproduce the work described in SpatialVLM to enhance the spatial reasoning of multimodal models.
With a pipeline of expert models, we can infer spatial relationships between objects in a scene to create VQA dataset for spatial reasoning.
- **Developed by:** remyx.ai
- **Model type:** MultiModal Model, Vision Language Model, LLaVA
- **License:** Apache-2.0
- **Finetuned from model:** LLaVA
### Model Sources
- **Dataset:** [SpaceLLaVA](https://huggingface.co/datasets/remyxai/vqasynth_spacellava)
- **Repository:** [VQASynth](https://github.com/remyxai/VQASynth/tree/main)
- **Paper:** [SpatialVLM](https://arxiv.org/abs/2401.12168)
## Uses
Use this model to query spatial relationships between objects in a scene.
[](https://colab.research.google.com/drive/1WPE7Br5A5ERSij8BL1M22EoEMLVkD8EP?usp=sharing)
Try it on Discord: http://discord.gg/b2yGuCNpuC
## Deployment
`docker build -f Dockerfile -t spacellava-server:latest`
`docker run -it --rm --gpus all -p8000:8000 -p8001:8001 -p8002:8002 --shm-size 12G spacellava-server:latest`
`python3 client.py --image_path "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg" --prompt "What is the distance between the man in the red hat and the pallet of boxes?"`
## Citation
```
@article{chen2024spatialvlm,
title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
journal = {arXiv preprint arXiv:2401.12168},
year = {2024},
url = {https://arxiv.org/abs/2401.12168},
}
@misc{liu2023llava,
title={Visual Instruction Tuning},
author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
publisher={NeurIPS},
year={2023},
}
``` |