The field of AI-driven generative design and robotics is rapidly advancing, with a focus on developing innovative methods for generating interactive tools, 3D objects, and immersive environments. Researchers are exploring the use of large language models (LLMs) and vision-language models (VLMs) to create physically conforming 3D objects, tools, and scenes that can be used in various applications, including engineering design, robotics, and virtual reality.

One of the key trends in this area is the development of pipelines and frameworks that can generate high-quality, interactive, and compositional data for training and testing AI models. These pipelines are enabling researchers to create more realistic and effective simulations, which can be used to improve the performance of AI systems in a variety of tasks, including robotic manipulation and tool use.

Another significant area of research is the development of methods for generating photorealistic textures and 3D models that can be used in immersive environments, such as virtual reality. These methods are allowing researchers to create more realistic and engaging environments, which can be used in a variety of applications, including gaming, education, and training.

Noteworthy papers in this area include:

• LLM-to-Phy3D, which introduces a novel online black-box refinement loop that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly.
• ImmerseGen, which proposes a novel agent-guided framework for compact and photorealistic world modeling, and achieves superior photorealism, spatial coherence, and rendering efficiency compared to prior methods.
• RobotSmith, which leverages the implicit physical knowledge embedded in VLMs alongside physics simulations to design and use tools for robotic manipulation, and consistently outperforms strong baselines in terms of task success rate and overall performance.