README.md

---
license: llama3.1
---

# LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models

[**Paper**](https://arxiv.org/pdf/2411.09595) | [**Project Page**](https://research.nvidia.com/labs/toronto-ai/LLaMA-Mesh/)

Pre-trained model weights of LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models


[Zhengyi Wang](https://thuwzy.github.io/), [Jonathan Lorraine](https://www.jonlorraine.com/), [Yikai Wang](https://yikaiw.github.io/), [Hang Su](https://www.suhangss.me/), [Jun Zhu](https://ml.cs.tsinghua.edu.cn/~jun/index.shtml), [Sanja Fidler](https://www.cs.utoronto.ca/~fidler/), [Xiaohui Zeng](https://www.cs.utoronto.ca/~xiaohui/)


Abstract: *This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. This offers key advantages of (1) leveraging spatial knowledge already embedded in LLMs, derived from textual sources like 3D tutorials, and (2) enabling conversational 3D generation and mesh understanding. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. To address this, we introduce LLaMA-Mesh, a novel approach that represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. We construct a supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate 3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs as required, and (3) understand and interpret 3D meshes. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format, effectively unifying the 3D and text modalities. LLaMA-Mesh achieves mesh generation quality on par with models trained from scratch while maintaining strong text generation performance.*


<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/634e15aec1ce28f1de91c470/CwSCmyJizQderIYC8CaJ4.mp4"></video>

## Method

Overview of our method. LLaMA-Mesh unifies text and 3D mesh in a uniform format by representing the numerical values of vertex coordinates and face definitions of a 3D mesh as plain text. Our model is trained using text and 3D interleaved data in an end-to-end manner. Therefore, our model can generate both text and 3D meshes in a unified model.

![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/634e15aec1ce28f1de91c470/0DzHXhoxonG5ZMeTysA6s.jpeg)

### Model Developer: Base model weight is from Meta. Finetuned by Nvidia

## Third-Party Community Consideration:
The base model, Llama 3.1, is not owned or developed by NVIDIA. NVIDIA has fine-tuned this model to meet specific application and use-case requirements. For more details about the base model, see the [Llama 3.1 Model Card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md). 

## License/Terms of Use:
This model, Llama-Mesh, is distributed under the following licenses:
1. NSCLv1 License
The Llama-Mesh model is licensed under the NSCLv1 license, which allows non-commercial use only. For details, please refer to the LICENSE.txt file.
2. Llama 3.1 Community License Agreement
This model incorporates components of Llama 3.1 technology, which is licensed under the Llama 3.1 Community License Agreement. Redistribution and use of Llama 3.1 materials must comply with the terms of this agreement. See the LLAMA_LICENSE.txt file for full details.

## Attribution
This model is built with Llama 3.1 technology, as required by the Llama 3.1 Community License Agreement. The required attribution is: "Built with Llama".

## Reference(s):
Llama 3.1 [Github](https://github.com/meta-llama/llama-models/tree/main/models/llama3_1) 

## Model Architecture:
**Architecture Type:** Transformer
*Network Architecture:* Llama 3.1

## Input: 
**Input Type(s):** Text

**Input Format(s):** String 

**Input Parameters:** 1D

**Other Properties Related to Input:** Max token length 8k

## Output: 
**Output Type(s):** Text

**Output Format:** String

**Output Parameters:** 1D

**Other Properties Related to Output:** Max token length 8k

**Supported Hardware Microarchitecture Compatibility:**
* NVIDIA Ada

**Supported Operating System(s):**
* Linux

## Model Version(s):
Llama 3.1 8B mesh

# Training Dataset: 
Please refer to [Llama 3.1  Model Card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md) for information on Training, Testing, and Evaluation Datasets).

The data is curated through converting Objaverse mesh data into text string (in the format as vertex index, face index as string). The model is finetuned on the curated dataset with 32 GPU. 

[**Objaverse**](https://objaverse.allenai.org/explore/) 

**Data Collection Method by dataset**: Unknown

**Labeling Method by dataset**: Unknown 

**Properties:** We use 30k mesh data, which is a subset from the Objaverse. We filter the Objaverse dataset by the number of faces, and only keep the shape with the number of faces less than 500. They are saved as obj file format. 

**Dataset License(s):** The use of the dataset as a whole is licensed under the ODC-By v1.0 license. 

[**UltraChat**](https://github.com/thunlp/UltraChat)

**Data Collection Method by dataset**: Unknown

**Labeling Method by dataset**: Unknown 

## Inference:
**Engine**: Pytorch

**Test Hardware**: A100

## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.  

Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

## BibTeX

```bibtex
@misc{wang2024llamameshunifying3dmesh,
    title={LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models}, 
    author={Zhengyi Wang and Jonathan Lorraine and Yikai Wang and Hang Su and Jun Zhu and Sanja Fidler and Xiaohui Zeng},
    year={2024},
    eprint={2411.09595},
    archivePrefix={arXiv},
    primaryClass={cs.LG},
    url={https://arxiv.org/abs/2411.09595}, 
} 
```