Livepeer AI Video Compute SPE Pre Proposal

[huangkuan]

Hi community,

This is Kuan Huang. Some of you might know me and my cofounder CJ from the project Scout.cool that we worked on a few years ago. Scout was an early analytics platform on the Livepeer network. We also ran an orchestrator for 2 years during the early days of the network. After Scout was acquired by Uniswap last year, we spent some time working there, but have recently moved on. We are currently exploring ideas at the intersection of AI, web3 media and blockchain.

Doug and Yondon from the Livepeer team have been talking to us about building an awesome consumer frontend app to be ready in time to pilot the launch of AI video capabilities on the Livepeer network, and suggested that we take the lead on forming the SPE to help Livepeer push this vision through.

They’ll be working closely with us on this, but I wanted to share this early pre-proposal for the Livepeer AI Video Compute SPE, for feedback.

Mission

The mission of this SPE is to quickly validate the impact and potential of bringing AI based video compute jobs onto the Livepeer network.

Approach and Strategy

Livepeer Founder, Doug Petkanics, highlighted the approach to quickly bringing AI video compute onto the network, validating the Network’s cost effectiveness and reliability, and showcasing this new capability to the world in this tweet thread. We intend to mirror this approach.

• Fund and support the core development of bringing Generative AI Video and supporting upscaling and frame interpolation tasks onto the network. Fund and support the core development and node operations of getting a subnet running on the Livepeer network.
• Build a consumer app experience that will be available and launch in conjunction with the network capabilities.
• Capture and showcase the data to the world.
• Set the project up with sustainable development and processes to take the scope of this SPE and continue the work required to merge the demonstrated capabilities into the core network software, and open up the network for more and more job types.
  
  

The above diagram shows the roles of the frontend app, spiked nodes to support a subnet, and use of the mainnet network to settle payments. We aim to help enable this reality, including development of the polished frontend consumer app.

We look at this as a point-in-time SPE that will deliver on the initial validation of these capabilities in the next 4 months. After that point ideally the initial app succeeds and grows as its own sustainable and self-motivated business outside of the scope of further public goods funding requirements, and this SPE or others can evolve to meet the needs of productionisation and further public goods needs around the AI video track. The funding requests will come in a series of milestone based proposals so that the community can continue to buy into the effectiveness of this SPE (or others) to continue meeting the project’s public goods goals around AI Video Compute.

Expected Impact

If we’re successful at bringing AI video compute tasks onto the Livepeer network, paired with a great consumer app that demonstrates it, then we expect to see the following impacts:

• An order of magnitude increase of fees flowing through the network towards node operators. If infra credits are made available via a treasury grant to this SPE, and consumers are interested in playing with generative AI video through the consumer app to the extent that the credits fund their free usage, it’s not out of the question that during the 2-3 months of a subsidized credit program, network fees could increase up to 4x.
• A growth marketing campaign can be run to highlight that Livepeer now has AI video compute capabilities, and ideally, that the network is shown to be more cost effective for running AI inference than the public cloud GPU networks. This can open up Livepeer to a whole world of demand for GPU based compute.
• Inspiring more AI video startups or developers to join the Livepeer ecosystem.

Upcoming Milestones

There are a number of technical, network, and product milestones which we can share over time in this thread as we flesh those out. But at this point I think it is more helpful to share the different funding milestones and what they unlock.

1. Core development of network capabilities + the consumer app. The initial treasury proposal would be for LPT to allocate to the core development of the network capabilities, lead by Livepeer co-founder and community member, Yondon Fu, and core development of the frontend app by CJ and I. Eli, who will be doing the Catalyst integration and API, is already employed with Livepeer Inc. There would also be some buffer for discretionary grants or payments to other community contributors on dev + testing.
2. Infra credits. A subsequent treasury proposal would be made to allocate LPT that could be used to pay the fees into the Livepeer network, earned by O nodes performing the AI compute tasks, to subsidize the initial “free tier” of the consumer app. The goal is that we can demonstrate the power of the network to the market, collect data and learn about network performance and cost, and increase fees up to 4x on the network during this subsidized period.
3. O node subsidy/bootstrapping program. This future proposal needs more work. After design, an operational program could be run to give grants to Os who demonstrably are running effective hardware on the sub-network, helping with testing, benchmarking, etc. They may incur hardware upgrade costs or be dedicating the time of the GPUs for testing when they otherwise could be put to other uses, and its reasonable that a bootstrapping program would exist to help cover some of these costs.

Team Members and Roles

Kuan Huang - Frontend app lead and SPE lead. Past experience: Founder of Scout, Product at Uniswap, founder of Poncho (a consumer mobile app which had grown to millions of audience across mobile and web). Compensated through the core dev proposal.

Chunxi Jiang (CJ) - Frontend app lead and SPE lead. Past experience: Formerly Founder of Scout.cool, Software Engineer at Uniswap, Visa, NYTimes. Compensated through the core dev proposal

Yondon Fu - Technical lead on AI capability and node implementation for core dev milestone 1. Compensated through the core dev proposal.

Sarah Armstrong - Ops member at Livepeer, who would be responsible for token operations, coordinating contracts needed for work engagements via Livepeer, Inc., payments to recipients, and potentially the subsidy/bootstrapping program ops. Sarah will also be responsible for continuity of the SPE beyond the listed milestones if future milestones are necessary or there are ongoing funding allocation requirements. No compensation from the proposal.

Doug Petkanics - SPE advisor and ultimately accountable for the success of introducing AI video compute onto the Livepeer network. No compensation from the proposal.

Eli Mallon - Catalyst engineer with Livepeer. No compensation from the proposal.

At least one future fulltime dev initially compensated through core dev proposal for a ~4 month period responsible for the intersection of AI and the Livepeer nodes, building upon the work from core dev milestone 1 in the productionization process.

Funding Requirements

Estimates:

• Core dev milestone 1: This initial proposal is for 25,000 LPT. We may transparently come back for more at future milestones with cost and budget justification.
• Infra credits: This is TBD for a future proposal. Potentially in the ballpark of 25000 LPT if the goal is 4x-ing fees on the Livepeer network over a 2 month period.
• O node bootstrapping program: TBD. Not enough is known yet for this milestone.

Transparency Commitments

We are committed to transparently sharing our plan, building in public with the community, showcasing the outcomes we can enable, and letting the community decide through future milestone funding requests whether we’re making worthy contributions or not. However we are also committed to moving quickly after funding is approved for a given milestone, so we’ll make decisions within the SPE and allocate funds accordingly to meet the goals, rather than pausing for community vote at every decision point.

• All code related to AI task capabilities, node integration, Catalyst will be developed in the open and be open source.
• Frontend application may not be entirely open source, as the point is to use it to validate Livepeer network capabilities and costs, but also potentially as future sustainable business that can drive usage to the network. Spiritually, we can open source or share openly any functional code that uses the network for AI inference, as a demonstration. But if we validate things like users paying for credits to expand their rate limits or other similar mechanisms, then it wouldn’t make sense that that billing/account related code is open source.
• The community will have transparency into each milestone’s goals and outcomes, but not internal transparency into allocation of funds within the core development milestone, as people have a right to private compensation data. In any subsequent treasury proposals related to core development, the community could use the observable outcomes from previous milestones to make a judgment about the worthiness of future allocations.
• In the subsequent milestones of infra credits, funds will be transparently observable on chain and used for network payments should there be enough demand. For any O node bootstrapping program, funds would be transparently allocated to recipients onchain.

Governance

• Released funds will be governed by the members listed above in the SPE. The community’s input is observed in the pre-proposal discussion, and during the vote at the particular milestones. But as funding is released, this team aims to move fast, and allocate the funds to accomplish the goals. The community can weigh in by supporting or not supporting future proposals from the SPE for future milestones based on their judgment on performance so far.

Future Funding Needs

See “Upcoming Milestones” section.

Thank you for considering this pre-proposal. Please share any feedback or questions here. We’re looking forward to moving quickly with the core team to bring AI video compute onto Livepeer as soon as possible.

[Karolak]

That looks great!
In my opinion, we shouldn’t subsidize hardware for orchestrators. The price of RTX 4080/4090 is within the reach of the majority of us. Taking the risk and investing in hardware should be the responsibility of each individual. Personal contribution will also motivate proactive participation. I believe everyone is excited enough to invest in a rig for the subnet out of pure curiosity. Taxing orchestrators to later redistribute funds is counterproductive. These funds should go entirely towards the development of AI integration. Individuals should take risks/investments autonomously according to their policy agreed with delegators, counting on future profits. Even renting cloud rtx 4090 is not a big cost. Otherwise, artificial inequalities will be created (known from socialism, this time it will definitely work). Option number 2 should be sufficient for a start. I will be more than happy to spin up a rig for your tests.

[yondon]

Here is a short update on initial work related to core dev milestone 1.

[Shann]

Very exciting to see this!

The goal of this treasury proposal appears to be simultaneously shipping both new network capabilities AND a consumer application using those capabilities – both currently listed as Milestone 1. I note that Milestones 2 and 3 are presented as future treasury proposals and so focus my comments here on Milestone 1…

There is a lot to unpack in Milestone 1 and I agree that the proposal will be strengthened by clearly articulating and distinguishing between the different deliverables and any sequencing therein. Below is some feedback I shared on the last community treasury call about areas where the proposal could be more explicit.

Developing new network capabilities

1. Technical Milestone: upgrading the protocol such that new AI job type requests can be sent, received and actioned permissionlessly between network actors [Orchestrators and Broadcasters].

This milestone alone is a significant undertaking for a 4 month period and (to compliment Yondon’s impressive initial technical update shared yesterday), it would be good to see a project roadmap for this.

2. Product Milestone: A developer-facing tool (eg hosted API or container such as Catalyst) that allows an(y) application to access the network’s AI processing capabilities.

An excellent developer facing product is needed to attract other AI developers and entrepreneurs to grow demand for this new network capability. Is the goal of this proposal to have a prototype of a tool to be further developed and tested in future treasury proposals by this or another SPE? At what point might there be exploration of whether this should be free or monetizable software?

A consumer application using new network capabilities

I think it’s worth distinguishing between 2 separate objectives that the proposal, as written, implies:

1. provide the core technical team with valuable user input to design the best solutions
2. launch an application with a good chance of becoming a successful business / breakout app that can drive demand on the network

I am fully supportive of a design partner approach to the work above so that technical and product decisions are made with end users needs first and foremost. To this end, I think it would be helpful to fund more than 1 design partner and come up with some criteria for what a good design partner looks like, eg already has 1. founders 2. a product, 3. users and (bonus points) 4. growth

If the focus of this proposal is more on the second objective, launching a breakout app, realistic and impactful milestones for this should also be articulated, for example:

1. Research. It would be great to see - either as part of this proposal or as one of the first milestones - some market research / evidenced-based hypotheses for the types of AI video jobs that represent demand growth opportunities for this new type of network “supply”.

2. Users. Is the intention for this application to have demand at launch in 4 months time? How will this initial user group be identified and engaged?

3. Business model. how might this app make money

4. Prototype, etc…

^ These are just some of the examples of what this SPE should include in its proposed roadmap if it intends to launch a successful consumer application in 4 months time

[huangkuan]

I will answer the questions which are related to the consumer app.

For milestone 1, the new AI capability will likely focus on text2video, image2video (couple seconds video output) generic job types: Something along the lines of THIS. There probably won’t be real time live video generation involved either. @yondon Feel free to chime in here.

The 1st priority of the consumer app is to demonstrate the new AI video capabilities and show prospective developers how to build apps on top of the new API(s).

The 2nd priority is to build an app that has some level of viralness and a decent chance to become a self sustained business. This could drive recurring demands/revenues to the network.

We already have three product concepts under discussion:

• Taking a selfie overlayed with stable diffusion technology
• Creating ecards with short video backgrounds
• Designing a virtual wardrobe that leverages the stable diffusion feature

Whatever idea that we may end up building will need to be filtered by the priorities that I have mentioned and the initial video capabilities that the network would be able to offer. And the unit cost of video generation is important as well.

[dob]

There have been a couple of questions related to the operations of the SPE now and on a go forward basis here and in the Discord thread that I’ll chime in to address:

4. SPE Function Overview : There’s a need for a more comprehensive understanding of the SPE’s function. While the pre-SPE forum post provides significant details, we still need clarity on the SPE’s operational dynamics post the initial development milestone. Key questions to consider include:

• How will the SPE evolve and function beyond the first development phase?
• Is there a plan to transition the SPE into a DAO (Decentralized Autonomous Organization)?
• Who are the key stakeholders and decision-makers involved in this phase?

The main goal of the SPE at this point is to achieve the core development milestone. This allows Livepeer to:

• Deliver on the technical capabilities to perform AI video compute within the context of the Livepeer network.
• Gather data about the cost effectiveness of the network in performing these tasks.
• Market this to the world with evidence

The above allows Livepeer to “grow” across many helpful vectors and contributor groups, including fees on the network, users, developing contributors, sources of capital, etc. The funding requested in milestone one allows for the committed core development resources, dev infra costs, additional vendors such as design, the ability to an additional longer term committed team member beyond who’s listed, and a bit of buffer in case of any delays or unknowns.

There are questions above about operations BEYOND milestone 1. To this I would say there are knowns and unknowns…

Knowns

• There’s continuity in the vision and responsibility for growing the project from myself.
• There’s continuity in the operations from our ops team member in terms managing the funds and handling administration of vendors, contracts, or future goals.
• The SPE creates a known vehicle for applying for further funds if warranted for further hiring, infra credits, roadmap steps or iterations.

Unknowns

• Will the consumer app be successful in its first iteration?
• Will it have a commercially viable path allowing it to raise independent funds and splinter off as one-of-many, or will it be viewed as a public good to the ecosystem while it iterates warranting further public goods funding for infra credits or even further development?
• What are the development requirements to go from the proof of concept early use cases to a scalable production architecture that allows devs to bring their own models onto the Livepeer network for AI inference? (This is an example of it just being early and there being a ton of day to day learning as we research, prototype, and develop the first app as a design partner).
• Will the proof of cost reduction actually be delivered under open market conditions on the Livepeer network?
• Will the market react to the marketing around this proof and will that lead to project growth warranting further funds allocation?

The knowns should give us confidence in the chance that this SPE can continue to be ONE impactful attempt at moving us quickly towards AI video compute and growth of Livepeer. The unknowns dictate that before anyone feels conviction about future funding from the treasury, they should observe the output and results that the SPE delivers! That’s part of the beauty of the community treasury and milestone-based funding: release a little early based on track record and reputation…release more later based on results!

As for key stakeholders and decision makers: As listed in the pre-proposal…Kuan+CJ on consumer app, Yondon on backend and network updates. Doug on overall accountability for introducing AI video compute to Livepeer. There is no plan to turn this SPE into a DAO - though through milestone based funding requests, the LP community will be making decisions as to whether to release funding for specific purposes into the SPE.

---

I am fully supportive of a design partner approach to the work above so that technical and product decisions are made with end users needs first and foremost. To this end, I think it would be helpful to fund more than 1 design partner and come up with some criteria for what a good design partner looks like, eg already has 1. founders 2. a product, 3. users and (bonus points) 4. growth

Yes, I want to reiterate that this SPE is ONE approach to accelerating growth in Livepeer, but it needn’t be the only approach by any means. If or when there is benefit to adding additional design partners to building on the early spike’s of AI video compute on the network, then there are a few options:

1. Join up with this SPE and propose another funding release to the treasury whereby you request funds to run a design partner recruitment/incentive program.
2. Do the same within the existing grants program (which can be viewed as its own SPE)
3. Propose another lightweight SPE focused on AI GTM or Growth

The reason this initial SPE proposal suggested one design partner (DP) (who’s actually leading the proposal which is great!) is because it’s hard to recruit DPs when there is no product and no proof of the eventual ability to meet that DP’s needs. It requires the DP to
be willing to commit time to learn, to be aligned with the ecosystem’s growth, and to build WITH Livepeer, rather than just being a potential user waiting for Livepeer to deliver. Easier said than done, and hence why the earliest DPs deserve some LPT based compensation for the months of their fulltime work they’re committing.

---

Anyway, hope this is helpful insight. In short, I would summarize as: Don’t get too hung up on the long term path for the SPE and consumer app. Instead, try to evaluate whether this group + strategy can move fast together to hit the proof-point milestones of introducing real AI video compute on the Livepeer network. We’ll all learn from there, and can evaluate and vote on funding for next steps based on the learnings and results.

[obodur]

In my humble opinion, once there is evidence that Livepeer is less costly (compared to clouds) when performing AI with its open market, then market, apps, dapps will jump on it. Money/Funding will not be an issue anymore. At all.
But to see that definitive evidence, a lot needs to happen. Because not only the AI infrastructure and testing app should be made ready but also it should be deployed and tested on the open O market. To see the pricing, right? To be able to compare. Time is needed. Maybe a lot of time.