Abstract

In this paper we study the problem of efficiently rendering images for embodied AI training workloads, where agent training involves rendering millions to billions of independent, low-resolution frames, often with simple lighting and shading, that serve as the agent's observations of the world. To enable high-throughput training from images, we design a flexible, batch-mode rendering interface that allows state-of-the-art GPU-accelerated batch world simulators to efficiently communicate with high-performance rendering backends. Using this interface we architect and compare two high-performance renderers: one based on the GPU hardware-accelerated graphics pipeline and a second based on a GPU software implementation of ray tracing. To evaluate these renderers and encourage further research by the graphics community in this area, we build a rendering benchmark for this under-explored regime. We find that the ray tracing renderer outperforms the rasterization-based solution across the benchmark on a datacenter-class GPU, while also performing competitively in geometrically complex environments on a high-end consumer GPU. When tasked to render large batches of independent 128x128 images, the ray tracer can exceed 100,000 frames per second per GPU for simple scenes, and exceed 10,000 frames per second per GPU on geometrically complex scenes from the HSSD dataset.

Code

The code for the renderer is available here (specifically, you can find the ray tracing and rasterizer code in the linked directories). You can also find a benchmarking repository to reproduce the results of the paper here.

Visualization

Example batch renderer outputs of the high geometry HSSD scenes.

Citation

@article{rosenzweig24madronarenderer,
    title   = {High-Throughput Batch Rendering for Embodied AI},
    author  = {Luc Guy Rosenzweig and Brennan Shacklett and
               Warren Xia and 
               Kayvon Fatahalian},
    conference = {SIGGRAPH Asia 2024 Conference Papers},
    year    = {2024}
}