Project information

  • Company: ORamaVR
  • Categories: Generative AI, 3D Deep Learning, NLP
  • Main technologies: Python, PyTorch, NVIDIA NeMo, Weights & Biases (WandB), vLLM, HuggingFace

Summary

Mugiwara is an end-to-end training and evaluation pipeline designed to teach Transformer models how to generate 3D meshes from scratch.

Leveraging the OVR-Datasets infrastructure, the pipeline streams raw 3D meshes, converts their geometric data into token sequences, and feeds them into a Transformer model that learns 3D shape reconstruction and generation.

To ensure scalability and fast iteration, the system was built on the NVIDIA NeMo framework. I integrated Weights & Biases for rigorous model evaluation and tracking. Finally, the trained models were published to HuggingFace and optimized with vLLM to ensure highly efficient, low-latency inference.

Making Meshes Fit: Octree Tokenization

A transformer can only attend over a fixed context window, so the cost of representing a mesh directly limits how complex a mesh the model can generate. Working alongside a research scientist, I co-designed an octree-based tokenization scheme that cut sequence length from 9 tokens per face down to 3, one per vertex.

That is a threefold increase in the mesh complexity representable within the same context window. We then tested and validated triangle-strip encoding, which reduced the token count by a further 50-60% on top of that.

From Offline Batch to Interactive: 10x Inference

The research prototype generated meshes far too slowly to be usable in a content creation workflow. I rebuilt the test-time pipeline around vLLM with batching and memory optimizations, bringing single-mesh generation down from 5-10 minutes to 30-60 seconds, roughly a 10x improvement.

The point was not the multiplier but the threshold it crossed: generation moved out of being an offline batch job and into a range where someone can sit and iterate on a result.

Research Process

The work ran on a single 8×A100 node on Azure, tracked in Weights & Biases. I was the sole engineer on the workstream, running experiments, presenting results and setting the next round of work with a supervising research scientist.

Much of it came out of reading the mesh generation literature and implementing the ideas. Not everything survived contact: I explored NeRF and Gaussian splatting models as an alternative representation and rejected them, because marching-cubes reconstruction produced meshes far too heavy to be usable in VR.