Introduction
Four months ago the AI Video SPE set out with the mission of quickly validating the impact and potential of bringing AI Video compute jobs onto the Livepeer network. It laid out a plan, following this thread and this proposal, which highlighted how it would hit this validation milestone with a very focused initial approach around proving the cost and performance of new generative AI video capabilities on the network by partnering with an initial design partner sample application.
While incredibly powerful in its own right, demonstrating prompt-to-video as the first new form of compute on the network beyond transcoding, is but one small step in a long and exciting journey of leveraging AI on the path to becoming the world’s open video infrastructure - a path that can eventually include AI video workflow enhancements like optimized encoding, subtitle generations, and smart clipping, as well as additional generative AI models and pipelines contributed by users. The world’s open video infrastructure should ultimately enable any creative form of video compute that users want to leverage.
The SPE largely executed according to plan and has made tremendous progress, discovering positive surprises in some areas and exceeding the estimations of deliverables, while inevitably learning that other aspects of its mission will require more time and iteration. A big highlight was scaling the team and operations around the AI video initiative. Through the SPE entity and funding, they were able to ramp up a full group of contributors spanning core team, community contributors, node operators, part-time developers, and more to focus on bringing AI video compute to Livepeer.
Achieved Milestones
- Validated technical capabilities of network with release of Subnet
After months of hard shipping, we have a working AI subnet, processing inference compute jobs, which can be built on permissionlessly. Rick and Yondon have optimised the Discovery & Selection algorithms, key to the effective routing of jobs from different Gateways to Orchestrators. Eli has added key metrics to ensure we have greater transparency on completed jobs of the new subnet.
Core Contributors: Rick, Yondon, Eli, John, Brad
- First consumer apps successfully built on top with users
The first consumer apps have successfully built on top of the subnet. You can view them under the Showcase part of the documentation. Kuan and CJ provided fantastic support as core design partners, working closely with the backend team to ensure the first capabilities were added. Other contributors have also taken initiative to experiment with different pipelines (Stronk) and focusing on other use cases (Titan, Brad, and Livepeer Cloud). Another big positive was that the first applications showed how much easier and more likely it is to build independent applications on top of self-run gateways in the generative AI space than it was purely with transcoding job types.
Core Contributors: Kuan, CJ, Stronk, Titan, Brad, Livepeer Cloud SPE
- First pipelines integrated successfully
The first pipelines to be supported were integrated:: txt2img, img2img and img2vid. One key note is the recognition of how easy it is to support additional models and support users adding additional pipelines. New pipelines are already being explored by other contributors and will be integrated during the coming months. While the clients only support these pipelines for now, in the future we will explore the possibility for anyone to deploy their own.
Core Contributors: Yondon, Rick, Kuan, CJ
- Successfully onboarded ~15 Orchestrators to Subnet
The initial process of onboarding Orchestrators involved manually onboarding each one, successfully done by Rick and Yondon. Each of the Orchestrators provided invaluable feedback, which has now been integrated into clear documentation for existing top 100 Orchestrators to integrate under the Livepeer AI docs.
Core Contributors: Rick, Yondon, Pon, Stronk, Titan, Brad ad Astra, John | Elite Encoder, Mike, SpeedyBird, Papa Bear, Sundara, bsinc2k, Jason
- Preparation for Public Launch at ETH Berlin
The first draft of documentation has already been written by Rick, with support from the Orchestrator community. The plan is now to publicly launch the Subnet during Berlin Blockchain Week, where we are hosting an event focused on generative AI and the new subnet capabilities. If you are in the city, please reach out to attend!
Core Contributors: Rick, Ecosystem Team
Network Impact
Though we are still in the early days of network validation, we have some very promising signs of the network capabilities:
- 15 AI node Orchestrators actively performing jobs
- 8,000+ text-to-image jobs completed
- 5,370+ image-to-video transformations done
- 15,500+ payment tickets issued
- 170+ winning AI tickets paid out, equivalent to 0.155 ETH (~$502)
Key Learnings
- Demand generation will require viral products
We made some fantastic progress with Tsunameme and other applications built on top of the network. However, demand generation of the scale we need requires in-built virality to the product that requires many iterations and constant user testing. We are confident that moving forward we can sponsor key products, including Tsunameme, that can harness this potential.
- Scaling supply to match demand is a challenge
We want to ensure that the existing Orchestrators are at near full capacity and that their GPUs are optimized to process jobs relevant to their processing power. We have already built in failover capabilities to ensure the liveness of the network. Now the question is how do we improve the hardware and capacity of the subnet whilst ensuring there are not idle GPUs.
- Developers want to experiment with open and decentralized AI
The good news is that many developers are interested in utilizing compute networks for AI inference jobs, alongside fine-tuning and training, over standard cloud providers. There is a strong open-source movement for models with communities such as Hugging Face providing a bright future to compete with closed models.
- Supporting independent builders and Integrations with existing products are likely paths to increase demand
Even though the Livepeer ecosystem should nurture projects that emerge from within, we are also entering a phase where we should look outwards for design partners who are committed to innovating with this technology in their own self motivated ways, or already have large user bases. Integrating these products will enable us to make larger leaps in demand to see the order of magnitude (10x, 100x, 1000x) increase in fees that we are all working towards.
- Demonstrating Livepeer’s AI capabilities to the outside world is key
Despite great progress during the AI SPE, we have not amplified our progress to the outside world as much as we potentially could have. We can now look to work with the Ecosystem team to run a growth marketing campaign to highlight that Livepeer now has AI video compute capabilities, and ideally, that the network is shown to be more cost effective for running AI inference than the public cloud GPU networks. This can open up Livepeer to a whole world of demand for GPU based compute.
Budget Allocation
The first SPE received 25K LPT in funding via the Livepeer Treasury. Below is a snapshot of the funds remaining in the SPE today (see SPE wallet on Arbitrum and Ethereum):
| Asset | Current Balance |
|---|---|
| LPT | 9,480 |
| USDC | 45,995 |
| ETH | 1.005 |
Fund Allocation Overview:
The SPE distributed the initial funds into four categories
- Contributor payments: payments made in USD and LPT to paid SPE contributors;
- Infrastructure: resources to support the subnet infrastructure;
- Operations: ETH for gas and funding the subnet broadcaster;
- Community engagement: organizing events like the community GIF competition.
Spending Breakdown (as of 3 May 2024)
| Category | Amount | Asset |
|---|---|---|
| SPE contributor payments | 7,500 | LPT |
| SPE contributor payments | $71,284 | USD |
| Infrastructure | $2,937 | USD |
| Ops: ETH for gas and Broadcaster | $4,884 | USD |
| Community Engagement | 20 | LPT |
| Total | 7,520 | LPT |
| Total | $79,104 | USD |
In total, the SPE spent 7,520 LPT and $79,104 USD as of May 3. Additional expenditures are anticipated, including 2,500 LPT and approximately $20,250 USD for contributor and infrastructure-related costs, expected to be finalized by May 15.
Forecasted Remaining Funds (as of 15 May 2024)
| Remaining payments (through May 15) | Amount | Asset |
|---|---|---|
| SPE contributor payments | 2,500 | LPT |
| SPE contributor and infrastructure payments | 20,250 | USD |
| Expected Remaining Balances on May 15th | 6,980 | 25,745 |
At the conclusion of the first SPE, the remaining funds are projected to include approximately 6,980 LPT and $25,745 USDC. It is proposed that these remaining balances be rolled over into Milestone 2 of the SPE (see below) to contribute towards the same mission with an updated set of deliverables.
Conclusion
The first stage of the SPE has shown that the Livepeer network potentially has a very bright future when it comes to AI. We have a strong set of Orchestrators who have shown great willingness to contribute to the future development of the network, and have attracted builders who are aligned with our mission of decentralized compute at an affordable rate.
The next steps are to secure the next round of treasury funding required to go from a subnet with technical capabilities and early demonstration apps to launching a mainnet with a series of successful applications that prove the quality and cost effectiveness of Livepeer’s network for handling AI video workloads. You can read the proposal for the next round of funding - with milestones, deliverables, team, budget, timelines, and more, as it’s posted in the coming days. I will link to it in this retro as it comes available.
It is an exciting time to be part of the Livepeer community.
