Earlier this month, NVIDIA announced the beta release of Omniverse , a platform on which developers and creators can create Metaverse apps. In this way, the company has aligned its future with the vision of the meta universe, and the new platform allows users to create “digital twins” to simulate the real world.
One such step toward realizing such a dream, which will help users visualize a high-resolution 3D model for any 2D image input or text prompt, is Magic3D . Magic3D, recently released by NVIDIA researchers, is a text-to-3D synthesis model that creates high-quality 3D grid models.
The model is a response to Google’s DreamFusion , in which the team used a pre-trained text-to-image diffusion model to get around the inability to have large-scale labeled 3D datasets, to optimize Neural Radiation Fields (NeRF). Magic3D addresses two of DreamFusion’s limitations – the extremely slow NeRF optimization and the low-resolution image space control in NeRF.
The model is based on a coarse-to-fine strategy that uses both low- and high-resolution diffusion before learning a three-dimensional representation of the target image. As a result, it can produce high-quality 3D mesh models in 40 minutes, on average twice as fast as DreamFusion, while providing eight times higher resolution supervisions.
NVIDIA uses a two-stage optimization structure to achieve fast, high-quality 3D output to text cues.
The first step in this process is to derive a coarse model using low-resolution pre-diffusion and optimize neural field representations (color, density, and normal fields). In the second step, the textured 3D mesh is differentially extracted from the density and color fields of the coarse model.
The output is then tuned using a high-resolution latent diffusion model, which generates high-quality 3D meshes with detailed textures after optimization.
The model also allows for quick editing. That is, given a rough model generated from a basic textual cue, parts of the text can be altered by fine-tuning the NeRF and 3D mesh models to produce an edited, high-resolution 3D mesh model.
In addition, Magic3D also has room for other editing capabilities in which for a given input image by fine-tuning the propagation model with DreamBooth and optimizing the 3D models with given cues ensures that the object in the rendered 3D image carries maximum fidelity to the object of the input image.
Leveraging eDiffi, NVIDIA’s text-to-picture stylistic rendering model, the input image can also be converted into the style of the output 3D model.
NVIDIA Corporation, known for its hardware prowess, is firmly entrenched on the generative AI front, even in the face of relentless competition from major technology companies such as Microsoft, Google and Meta, which are actively working to integrate their platforms with advanced AI model technologies.