AI model improvements for higher-quality textures in Unity Muse

At Unite 2023, we introduced the world to Unity Muse, a suite of capabilities to help you explore, ideate, and iterate with greater ease in the Unity Editor. One of these capabilities is Texture. Muse’s texture capability transforms natural language and visual inputs into PBR-enabled materials that can be applied directly to surfaces and objects in your project.

In our blog “Responsible AI and enhanced model training at Unity,” we shared how Muse’s foundational model for textures, Photo-Real-Unity-Texture-1, was responsibly built and trained so that outputs are safe to use and respectful of others’ copyrights. Specifically, we detailed how we pioneered a bespoke diffusion model trained from scratch on proprietary data that is Unity-owned or licensed.

We remain committed to responsibly building AI in service of our creators and today, so we are launching the next generation of this model, Photo-Real-Unity-Texture-2.

In the latest version of our model, we focused on improving material types that commonly occur in games. Materials such as wood, bricks, concrete, leather, metals, gravel, soil, and many others have all had significant improvements. Prompt-based color consistency has also improved, and an upcoming release of the Muse Texture package will include the ability to choose both a specific color and specific pattern at the same time. 

In addition, our model has significant improvements related to the creation of heightmaps. Heightmaps are now 16 bit by default, and the model is much more capable of producing believable maps for abstract concepts in many cases.

Photo-Real-Unity-Texture-2 has been implemented into Unity Muse, so all Muse users can take advantage of these improved textures today.

These side-by-side comparisons show the difference in quality between Photo-Real-Unity-Texture 1 and Photo-Real-Unity Texture 2.

The images on the left are from our previous model, and images on the right are from our new one.

Depth is a critical aspect of achieving high-quality interaction with light. The side-by-side comparisons show the difference in quality of the height maps and normals between Photo-Real-Unity-Texture-1 versus Photo-Real-Unity-Texture-2.

The images on the left are from our previous model, while images on the right are from our new model.

We will continue improving Muse Texture’s output quality in the most responsible way possible. Other areas for improvement in our current roadmap include Editor integration, artistic control, and better tools for education and onboarding. 

Editor integration

You’ve told us that deeply integrating Muse Texture into your Editor workflow is paramount to your success. We will continue doing so and aim to create as seamless an experience as possible with new features such as quick in-scene variation, and by making it easier to bring our material maps into other shaders.

Artistic control

The models will continue improving, but we learned that many of you want and need more artistic control to achieve your desired results. Additional features that improve artistic control will enable you to achieve your vision more quickly and easily. We plan to further improve color control and consistency, add the ability to convert photos or reference images to materials, and allow you to control texture stylization using reference assets.

Education and onboarding

Generative AI is a new technology. We will focus on better onboarding and learning materials to help you leverage the tool’s features for deep artistic control. We will focus on shortening your journey to help you find success as quickly as possible. 

Try Unity Muse free for your first 15 days to experience how AI capabilities allow you to prototype faster in the Unity Editor. If you have questions, check out our AI FAQ, or visit our dedicated Unity Muse Discussions to chat with us directly.