DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning

Ruowen Zhao*1,3, Junliang Ye*1,3, Zhengyi Wang*1,3,
Guangce Liu3, Yiwen Chen2, Yikai Wang1, Jun Zhu1,3
1Tsinghua University, 2Nanyang Technological University, 3ShengShu
(*Equal Contribution)
arXiv Code Checkpoint

Demo Video

All of the meshes above are generated by DeepMesh. DeepMesh can generate high-quality meshes conditioned on the given point cloud by auto-regressive transformer.

Point-cloud Conditioned Mesh Generation

DeepMesh creates the mesh on the right from the point cloud on the left. Drag with the left mouse button to change the view, right mouse button to move the mesh.

Animation of Mesh Generation

The following video shows an animation of the mesh generation process. We generate all faces of mesh sequentially.

Abstract

Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality.

Method

MY ALT TEXT

Fig. the pipeline of DeepMesh. DeepMesh is an auto-regressive transformer composed of both self-attention and cross-attention layers. The model is pre-trained on discrete mesh tokens generated by our improved tokenization algorithm. To further enhance the quality of results, we propose a scoring standard that combines 3D metrics with human evaluation. With this standard, we annotate 5,000 preference pairs and then post-train the model with DPO to align its outputs with human preferences.

BibTeX

                
                    @misc{zhao2025deepmeshautoregressiveartistmeshcreation,
                        title={DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning}, 
                        author={Ruowen Zhao and Junliang Ye and Zhengyi Wang and Guangce Liu and Yiwen Chen and Yikai Wang and Jun Zhu},
                        year={2025},
                        eprint={2503.15265},
                        archivePrefix={arXiv},
                        primaryClass={cs.CV},
                        url={https://arxiv.org/abs/2503.15265}, 
                  }