Find the right AI model for your game with Unity and Hugging Face

We’re excited to share a new integration with Hugging Face and some product updates for Unity Sentis, our neural engine for bringing AI models into Unity. With Sentis, we aim to make AI models more accessible to help you power in-game features like object recognition, smart NPCs, graphics optimizations, and more.

However, given how many models are available, it’s often difficult to know which one fits your use case, ensure it’s reliable, and determine whether it works well with Unity. This is why we integrated with Hugging Face, the largest AI model hub in the world (470,000+ models as of this writing), to offer validated AI models for Unity Sentis that our team has tested to ensure they integrate seamlessly.

If you’d like to start with Unity Sentis right away, check out our blog overview, documentation, or dive into the community.

Going forward, we’ll continue collaborating with Hugging Face to provide high-quality, validated AI models for use with the Unity Engine. These models are optimized for easy implementation into Unity with the Sentis framework, tagged with “Unity Sentis” in the Hugging Face hub. We also share the corresponding ready-to-use C# file to perform inference in your game. The models are preconverted to the .sentis format to bypass the ONNX conversion step. Today, we’re offering 11 validated models that cover common use cases the community has asked for, like sentence similarity, speech to text, object detection, and more.

In order to build an expansive free and open-source set of models across all use cases, we encourage you to post personal projects of interest to the Hugging Face Hub using the Unity Sentis tag. 

Please reach out on the discussions forum for help adding a model or to share the link to your Hugging Face uploads with the community.

Hugging Face also works well with Unity, even if your use case is not conducive to running AI model inferences locally with Sentis. You can use the Hugging Face Unity API package to host your model on the Hugging Face Inference API, which is a free API for rapid prototyping, or you can use their Inference Endpoints, a paid API for production use.

Unity Sentis entered open beta as a prerelease package in 2023, and the team has been working hard to bring some stability improvements for early this year. The most recent release is focused on addressing issues identified by our developer community. We really appreciate everyone’s continued feedback to improve the package.

Unity Sentis will officially launch later this year as part of the Unity 6 release. Until then, Sentis will remain a prerelease package in Unity 2023 as we add consistent feature updates and fixes. There are also a number of exciting product updates planned.

It can be challenging to implement and optimize your model without a good visualization. To this end, we plan to create an intuitive graph viewer for AI models that allows for native visualization directly inside the Unity Editor. In addition, this will give you the ability to dispatch model nodes (operations) to different compute types (CPU vs GPU). Yay, graph-based visual optimization!

Performance is always a concern for game developers. Our plan is to integrate with hardware acceleration libraries and neural-specific chipsets, also known as neural processing units (NPUs), on platforms like Microsoft Direct ML, Apple Core ML/MPS Graph, Google NN API, and more to achieve faster inference on every device. What’s better, when inference is run on an NPU, it means the computation is offloaded from CPU/GPU to leave more budget for your game needs.

While many AI models can run on-device with Sentis, there are some cases where having a cloud inference is preferred. For instance, the model may only be available through a hosted service, or you need to maintain server authority, or you require faster performance on low-end devices. 

These cases can be solved for if you want to use standard Unity web requests with your API, or, as previously mentioned if you are using Hugging Face infrastructure you can use the Hugging Face Unity API. In the future, we plan to work with Hugging Face further to align their Unity API with the Sentis API to keep the local and cloud inference inputs and outputs consistent so it’s easier to prototype between native and cloud compute without modifying your C# code.

Unity Sentis is now available for free in open beta to all Unity developers operating on Unity 2021.3 or higher through the Package Manager. If you’re already working with Sentis and want to collaborate with our team, please fill out this form.