Embody Project Update
In this thread, I will explain the progress made on the Embody project since the last update. If you would like to listen to this thread or experience what has been built so far, you can have Lucy, Sarah, Harry, or Zach read it to you.
Most of my time over the last three months has been spent researching optimization techniques, creating proprietary 3D assets, adding new WebRTC and TCP hooks, and reworking important systems to match our long-term scope. Here’s why:
1. Optimization
At maximum level of detail for MetaHumans, optimization for their scene components becomes tricky. The mesh has over 80,000 vertices, the armature has approximately 800 bones, and we use particle simulations instead of hair cards for realistic hair movement. As you can imagine, these scene components are heavy—even without expensive rendering features like ray tracing, Lumen, etc.
My current approach:
-
Why are we computing the transforms of bones that aren’t moving?
-
Which post-processing effects will benefit us, and which are dead weight?
-
What logic should I write in C++ versus Blueprints, and how does it interact with the CPU versus the GPU?
Scalability and cost management:
Running one MetaHuman per GPU is expensive; running 20 of them is realistically scalable. In order for Embody to be a long-term success, we must increase cost efficiency. This means lowering GPU and CPU usage is mission-critical.
2. Creating Proprietary Assets
The intention behind creating proprietary assets stems from multiple considerations. DeFine and I have discussed the possibility of the game build eventually being open source. The only way we can legally do this, from what I understand, is for the assets to be proprietary, so we have the legal right to release them through Creative Commons.
All assets used outside of our proprietary 3D content are currently placeholders and would eventually need to be developed internally. By making the assets in-house, we also maintain the right to transform and modify them in any way we please. This allows us the ability to do things such as allowing companies to place their logo on the avatar’s clothing, or use Gemini prompts to change the materials.
By making the assets ourselves, we have full creative freedom, whereas purchased assets or generative assets can sometimes be restrictive in licensing terms. The beauty of proprietary 3D assets is that once they’re finished, they’re finished. They don’t require maintenance from developers, they can be easily modified with morph targets or parametric materials, and we have full control over their optimization.
3. Hooks
The Embody project provides hundreds of hooks that execute in-game logic either through C++ or Blueprint logic, most of it being Blueprint logic with C+±wrapped nodes. These hooks allow the developers to customize the avatar, spawn objects, apply post-processing effects, and essentially control the game from outside of the game.
Recent innovations:
-
The HLS media texture will play Daydream outputs. We even have a method that updates the Daydream prompts as the TTS output plays, resulting in backgrounds that match the context of the conversation. I’m working on creating a WHEP solution for blazing-fast latency between TTS output and Daydream output.
-
You can now change any material in the game with a prompt, removing my need to texture each skin individually. Goodbye, Substance Painter 3D—Gemini saves me almost $40 a month. If you want to be more creative, we can also paint on the mesh directly through the front end.
-
Every hairstyle available in MetaHuman Creator is now accessible at runtime.
-
MetaHumans now have proprietary clothing with multiple default skins to start from.
-
You can choose your avatar’s idle animation.
-
We have the ability to switch between TTS providers, including the ability to use our own models.
-
We have one-click backgrounds that spawn 3D environments with a single command (think film studio scale, not full worlds yet).
-
We can spawn lights into the scene and even animate their intensity and colors.
-
A narration mode has been added that prevents the blend shapes for lip sync from playing, creating a narrator effect.
4. Reworking Important Systems
After updating to Unreal Engine 5.7, I found myself in a difficult position. Epic Games changed the way MetaHumans worked entirely compared to 5.5. The MetaHuman Creator was no longer browser-based, assembling the MetaHuman overrides the Actor Blueprint rather than updating the current one, the material system utilized baked textures and was no longer parametric, I had to recreate Sarah because I wasn’t her original creator, and much more.
My solutions:
-
Creating duplicate MetaHuman Actor Blueprints to retain their logic and character class setup.
-
Creating my own material function that blends parametric materials with the new baked material system.
-
Creating a brand-new Sarah.
I also want to eventually provide users with the ability to explore open worlds and receive player input directly through the browser. So I switched the camera system from the camera being independent to being attached to the avatars with a spring arm. We can now switch between a playable mode and an idle mode. However, this feature likely won’t be expanded upon until other optimization and networking systems are enabled.
5. Conclusion
Look, I know progress can feel slow—believe me, it feels slow for me as well. Like any great artist or developer, sometimes I am ashamed of my progress and disappointed in myself. I wish I could produce more, but what I have built so far is a fantastic foundation for something that could change the way people interact with artificial intelligence.
We have strengths, and we have weaknesses, just like any other project. As time goes on, those weaknesses will fade. Animation will get better, customization will be limitless, the benefits of a 3D world will become more apparent, and much more. I just hope that the community continues to support our project. We have all worked very hard to get here.
I want to thank everyone for following along and supporting us, as well as the current and previous teams that helped kickstart embodied artificial intelligence. To DeFine, Titan, Phoenix, and all of the pioneers before us—thank you.
Note: This thread was written by hand but formatted into a markdown by AI.