New tool lets users repair AI-generated 3D models, then fabricate them just the way they want

“What you see is what you get” is a guiding principle for many software engineers — create programs where the content you’re editing looks the same as the final product. But when you’re using generative artificial intelligence systems to 3D print, say, a mug, you’ll likely get a cup that can’t hold your coffee. Why is that?

The issue is that AI models understand how an object should look, but not how it works, leading to impractical designs that undermine an item’s intended use. Even if you want to fix these errors, the models are typically hard to edit, especially for users new to 3D design.

A new approach called “InstructMesh” makes it much easier to design and print household items, accessories, and robots that work in the real world. The design software, which was developed by researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Google, and Northeastern University, can be prompted to generate a 3D design for a pair of glasses, for instance, and users can then highlight specific parts of the blueprint they’d like refined before 3D printing. It’s an AI-driven interface designed to understand how these designs should look and which edits experts and novices alike want to make, helping them create the objects they actually want to see.

CSAIL researchers used InstructMesh to put a personalized, creative spin on otherwise regular items. For example, the tool produced a mug that appears to be enveloped by a dragon, with its tail being the handle. It also fabricated a shiny blue whistle resembling a shell and a pair of glasses with butterfly wings spreading out just above the lens. Getting even more creative, it made an octopus-like dispenser, with liquids flowing out of each tentacle to distribute drinks into several cups at once.