Novices Fix AI-Generated 3D Design Flaws With InstructMesh in Research Tests

A convincing AI-generated mug is not necessarily a usable cup. In an October 1 account, MIT detailed InstructMesh, a research tool that lets people repair generated 3D designs before printing. Researchers reported that novices identified and fixed flaws around 90% of the time, with an expert reviewing their work.

The project, developed by researchers at MIT CSAIL, Google and Northeastern University, addresses a gap between appearance and function. Generative systems can produce plausible-looking shapes whose geometry—their actual structure—compromises their intended use after fabrication. InstructMesh puts a repair stage between that first output and the physical object.

The underlying paper was submitted to arXiv on August 28; this is a research project, not a newly established commercial rollout. MIT’s account adds concrete examples and a numerical description of the novice repair results. The researchers plan to present the work at the ACM Symposium on User Interface Software and Technology in November.

Repair the selected part, not the whole idea

InstructMesh combines Microsoft’s TRELLIS, which generates 3D models from text and images, with GPT-4. Users highlight a region of the design and describe the change they want. They can also use sliders for more precise adjustments, including enlarging a part or extending it outward.

The paper describes targeted operations such as opening or sealing voids and adjusting local thickness. Those are changes to an object’s structure, rather than simply its visual style. The interface is designed to let people make fabrication-relevant corrections without needing expert skills in conventional 3D modeling software.

Edits happen in the generator’s intermediate latent representation: the internal form it uses while constructing a model. InstructMesh translates the requested correction into a geometric change that the user can evaluate and approve. Human judgment remains part of the workflow; the system does not replace that approval step.

The flaw rate and the repair result measure different things

To examine the problem, the team had TRELLIS recreate popular designs from Thingiverse, a platform for 3D-printable models. Nearly 80% of those generated models had structural flaws of some kind. That figure describes the tested recreations, not a measured failure rate for every AI-generated object.

Nearly 80%Generated models with structural flaws

MIT says nearly 80% of TRELLIS-generated recreations of popular Thingiverse designs were structurally flawed.

Around 90%Novice identification and repair

Novices identified and fixed issues around 90% of the time, according to the researchers’ expert-reviewed results.

Novices were then asked to find and correct the issues using InstructMesh. The roughly 90% result concerns that identification-and-repair task, as judged by an expert. It is not a before-and-after comparison with the 80% figure, and should not be read as a guarantee of printed-object durability.

The paper describes two user studies and reports a preference for combining natural-language input with sliders. MIT says users made objects resembling phone stands and vases and found the tool easy to use. The combination gives them a way to express an idea in words, then make a more controlled adjustment.

Working examples, with physics still on the roadmap

The team’s demonstrations included a dragon-themed mug with a tail-shaped handle and an octopus-like dispenser that sends liquid through its tentacles into several cups. Researchers also made a shrimp-shaped enclosure for a small motorized bristle bot. These examples extend the project beyond decorative shapes to objects with intended functions.

A proposed next step would bring physical behavior into the design process. Lead author Faraz Faruqi says InstructMesh may incorporate physics simulations to explore whether a bowl would break when dropped and which materials would work best. That remains a possible extension, distinct from the demonstrated ability to help users repair geometry.