Open-ended Example-Based Shape Modeling

Our group explored the challenging field of open-ended example-based 3D shape generation and synthesis. We developed an autoregressive transformer decoder that leverages the extrinsic (GMM) and intrinsic (surface geometry) representations to automatically complete the part selection process based on a small set of example shapes.

Left: input example shapes

Right: Generated shapes

Training the decoder requires enormous 3D data. I solved this limitation by constructing a dataset of meaningful combinations of example shapes and their corresponding “ideal” parts to compose. Because high-quality 3D mesh data is challenging to obtain, I used a cascaded diffusion model to complete and reconstruct part-level shapes. I split the existing shapes into parts and reconstructed complete shapes for each part, creating a meaningful training dataset containing more than 60,000 3D shapes.

The example shapes are partially identical to the ground truth while maintaining diversity across each other. Partially identical completion maintains intrinsic and extrinsic features of the incomplete shape by using preserved priors in the diffusion model’s denoising process, ensuring consistent surface geometry. Additionally, introducing a diversity of example shapes measured by Bhattacharyya distance and supported by introducing bias in the diffusion model enhances the transformer decoder’s generalization ability.