Call for Transcoder Feedback - Streamflow Protocol Update Proposal Draft

[dob]

The Livepeer core team has been working hard on a proposal for the next major Livepeer release that makes the network:

• More reliable
• Much more cost effective
• More performant
• More scalable

The codename for this proposal is Streamflow. It is currently in draft, and I would like to invite the network’s active and aspiring transcoders to review some of the ideas, provide feedback, and ask questions:

Review the early draft of the Streamflow proposal here.

Streamflow Highlights

Streamflow introduces some new and changing concepts to Livepeer. Some of the more noticeable ones to all users will be:

• Elimination of the limit of number of active transcoders. Currently this is arbitrarily 15, but in Streamflow, anyone can join the network as an active transcoder with any amount of stake.

• Offchain job negotiation rather than on chain job assignment - major improvements for video developers using Livepeer, as they can now work with multiple nodes for redundancy or switch quickly in the case that the transcoder they are working with becomes unavailable.

• Payments via probabilistic micropayments (PMs) - this means that nodes won’t need to do 5 transactions on chain to get paid for each job, but instead can just cash in their earnings whenever it is affordable to do so on the Ethereum blockchain. This is a major cost reducer, and can make the Livepeer network very cheap to use for developers regardless of the fluctuations in price of Ethereum gas.

• The proposal also introduces a new role called Orchestrator, which coordinates work amongst a pool of transcoders for further reliability and better devOps setups, enabling GPU transcoding pools.

While the amount of fees flowing through mainnet are limited at the moment due to some of the costs of using the Ethereum blockchain, this proposal will unlock all the early pilot users to be able to use Livepeer with high reliability and cost effectiveness. Fees will begin flowing from the early pilot users to the node operators and delegators.

At this stage, it would be great to get feedback from node operators, or anyone in the community who is interested. As the proposal moves beyond draft, it will be shared more publicly for peer review, analysis, and audits on any eventual implementation.

Thanks, and please ask questions here!

[deboot]

Some quick high level questions to kick things off:

1. What are the implications of the introduction of the new Orchestrator role for a) centralization, b) censorship-resistance?

2. Re: verification, the proposal says that “Broadcasters are responsible to verify received transcoded segments”. Can you elaborate some more on how you see this working in practice, as right now it would seem to introduce even more complexity for a broadcaster to use the service.

3. Also relating to validation, what are the pros / cons of effectively outsourcing this verification to a different O / T, e.g. occasionally sending segments to be transcoded via 2 different channels, and comparing the results?

4. How do these Livepeer-specific scaling mechanisms fit with broader work being done to scale Ethereum’s network? Are there potential synergies, or is this work largely independent of that being done by the rest of the community?

5. What are the plans for growing the fees available for Orchestrators and Transcoders? Can you share more information about the “early pilot users”, and what we consider to be the unique selling points of Livepeer’s Transcoding over other options?

[dob]

Good questions.

1. What are the implications of the introduction of the new Orchestrator role for a) centralization, b) censorship-resistance?

This should be a powerful decentralizing force. It dramatically lowers the barrier to entry for anyone to be able to compete on pricing and quality of service, without needing to meet a “top 15 stake” requirement in order to be active on the network.

As for censorship resistance, any O can of course refuse to do work on certain content (should they choose to monitor/moderate), however with an abundance of O’s offering different services, there should generally be a fallback option for B’s. In fact, I think O’s could optionally advertise exactly this feature (and maybe charge a premium for it) if they philosophically stood for that.

2. Re: verification, the proposal says that “Broadcasters are responsible to verify received transcoded segments”. Can you elaborate some more on how you see this working in practice, as right now it would seem to introduce even more complexity for a broadcaster to use the service.

B’s node by default could just re-encode some randomly selected segments (say 1 out of every 1000), and see if it matched what O returned to them. If so, all good. If not, the node would challenge the result on chain. No human interaction needs to be involved.

3. Also relating to validation, what are the pros / cons of effectively outsourcing this verification to a different O / T, e.g. occasionally sending segments to be transcoded via 2 different channels, and comparing the results?

This seems like a good strategy, especially if you want redundancy anyway. Though from a game theory perspective you’d have to worry about the two O’s you sent it to colluding with one another.

If B randomizes it’s “checker” O’s, and takes their work history and probabilities into account, then it’s probably an ok solution but not perfectly secure. In reality I see these “strategies” implemented as open source modules such that B’s can choose what strategy they want to use depending upon their use case and preferences.

4. How do these Livepeer-specific scaling mechanisms fit with broader work being done to scale Ethereum’s network? Are there potential synergies, or is this work largely independent of that being done by the rest of the community?

The Probabilistic Micropayments piece is an area of active collaboration with the Ethereum community, and can likely be used by many projects.

As for the main Ethereum scaling roadmap - Serenity, Plasma, DAppChains, Interoperability solutions, etc - we took the approach that we didn’t want to be blocked by any 3rd party future developments, where we’d have to wait for external parties to make progress before Livepeer could be useable. So this proposal works on the current Ethereum today.

5. What are the plans for growing the fees available for Orchestrators and Transcoders? Can you share more information about the “early pilot users”, and what we consider to be the unique selling points of Livepeer’s Transcoding over other options?

Livepeer is focused on talking to and learning from video applications that use transcoding at scale, are open to running on decentralized infrastructure, and subsidize the cost of their streaming (think user generated content applications rather than subscription premium content services).

The main value prop for many of them is that the network can be far cheaper than the cloud providers they are using, though of course others have different value props with regards to decentralization.

[dob]

I got a bunch of questions during today’s call with some of the transcoders on the network. Here are some responses:

How do you ensure that if verification fails the amount of the refund matches up with the amount paid for the work since its probabilistic?

The penalty to an orchestrator for failing a verification should far exceed the amount that was paid to them during the time window in which work was being done incorrectly. So for example, they would be slashed a significant portion of their stake, which could be many thousands of LPT. Whereas even with a very infrequent check interval for verification, it would likely only be a few cents worth or transcoding that they got away with doing incorrectly.

Why probabilistic micropayments? What’s the motivation?

This is a big topic. I suggest you start by reading this post from Orchid Protocol for background. In short:

• Dramatic cost reduction for everyone - You as an orchestrator can determine the cost overhead of cashing payments on Ethereum, rather than be subjected to various overheads depending upon the price of Ethereum gas.

• You don’t need to pay the overhead to open a payment channel for each Broadcaster/Orchestrator relationship. This wouldn’t be worth it if the relationship was short lived for only a few segments of video.

• You don’t need to pay the overhead of releasing payments on chain for each stream (especially if it’s short, and wouldn’t be worth it). You only cash payments when you collect a certain amount that is definitely worth it.

How can you enable public transcoder pools? Or is being an orchestrator just about scaling/partnering with trusted private pools?

This is an open research area. Public transcoder pools can definitely exist, but they incur cost overhead of the O having to verify the work that T’s perform, either through redundancy or metrics based sampling.

One idea is to create a protocol with economic guarantees, where T’s put up stake to participate in a public pool. O doesn’t need to verify their work, but can penalize them if their work leads to slashing.

Further ideas and research are necessary.

Who chooses who in the network?

• Broadcasters get price/service quotes from Orchestrators and select to work with one or many. O can stop doing work at any time for B, and B should be resilient enough to just work with another O instantly. Failure is expected and the network can still provide reliability in the case of any failing O.

• O’s likely run their own T processes, though if they operate a public pool then the relationship between O and T would be mutual.

How much minimum stake is required to be an O?

There is no minimum. Anyone can be an O. But the clients will likely enforce a minimum by default to give B’s protection against their streams being tampered with. Alternate implementations could override this, or it could be overriden by configuration for other use cases or B’s who want to work with and select their own O’s regardless of stake. One example here would be a B that chooses to run it’s own O and pool of T’s on their own internal network - but then goes out to the open marketplace whenever their network has no additional capacity.

[0x88]

Nice… love it

[velvetdoctor]

I am a delegator in livepeer network and I have a few questions regarding Streamflow. Just FYI, I am a complete non-technical guy so some of these questions might seem amateur or silly but I thought of asking it anyway Sorry if I am wasting your time!

1. You mentioned, that the orchestrator could be slashed huge amount of stake deposit if he tried to cheat. So how could delegators know which orchestrators are cheating and which are genuine? Will delegators lose their LPT if the orchestrator to whom they delegated their LPT is cheating? Or is the slashed amount only taken from Orchestrator’s personal stake?
   If delegators are not losing anything, then is there a possibility that an orchestrator could divide his own LPT in to multiple delegator accounts and risk cheating as majority of his LPT are in the form of delegator accounts and hence safe? Will the slashed amount be always greater than the profit from cheating? How do you ensure that (especially when broadcasters check only 1 out of 1000 segments. Is that enough validation? Is there any possibility that an orchestrator could cheat some segments while doing other segments right)?

2. Will the following metrics be published on livepeer explorer or some other place so that delegators could check the following performance metrics of each orchestrator:
   How many times they are slashed
   How much work they did
   How much fees they earned in the last 24 hours
   How much fees was distributed to delegators
   what was the ratio of fees/LPT?

3. Will delegators be able to see all the services and locations registered by Orchestrators in service registry? Will that be available on livepeer explorer or in some other place?

4. How do the protocol know whether the mistake happenned is from Broadcaster side (not paying amount properly) or from Orchestrator side (not performing work properly)? How do you ensure that Orchestrator is not getting slashed because of Boradcaster’s mistake?

5. What are some of the steps that delegators should take to ensure that they are delegating to the right orchestrator?

6. If two or three big Broadcasters keep working with only two or three trusted orchestrators, and if delegators see that most of the fees is going to those two or three orchestrators and hence delegate their LPT to them, then won’t the network be centralized? How can you avoid this scenario (especially in the early days)?

7. If two or three orchestrators (who are early investors in Livepeer) have more LPT tokens than other orchestrators, have 0% fee share and provide services at cheaper rates than other smaller orchestrators, why would any broadcaster worry about whether this orchestrator is sharing their fees or not? In that case won’t the broadcasters keep sending work to the same big orchestrator as long as the work is performed genuinely? In short, why should a big orchestrator (who has his own large LPT stack and efficient hardware to provide services cheaply) worry about fee sharing at all?

8. what is the approximate timeline for Streamflow to go live? 6 months? 1 Year?

9. Do you have any broadcasters ready to start broadcasting on livepeer once Streamflow goes live? If yes, what is the volume (in terms of USD) that you are expecting?

10. Are there any transcoders in livepeer network that are currently earning fees (ETH) by doing transcoding job?

11. Is there any waiting time between broadcaster paying the fees, orchestrator receiving the fees and distributing that fees to delegators? Will that distribution be done at the end of each round or is it done at orchestrator’s discretion?

[dob]

These are great questions. I’ll try to give some quick responses.

1. Will delegators also be slashed/how will they know if their O is behaving correctly…

Yes, D(elegators) will also be slashed proportionally along with the O. O’s history should be visible on chain, but also through tools like the explorer, so that D’s can evaluate their level of risk. Remember that no one should ever be slashed unless they do something proactively malicious.

Of course the other scenario is that they are accidentally slashed due to a bug - and even worse is if someone malicious can trigger this bug remotely by crafting a certain input segment. We are thinking about “rate limiting” solutions in which harm due to accidental slashing could potentially be limited, especially in the early days of the network.

The 1-out-of-1000 is just an example parameter, and far more segments could be checked, or there could be built in checking via redundancy. The average time to detect a slash should be bounded with high probability, and since it’s likely only a few Ethereum blocks, (or couple minutes) worth of encoding - this likely costs pennies worth of work, but any slash amount that represented a percentage of a significant amount of staked LPT, would be a harsher penalty.

2. Metrics…

Yes, all those metrics you suggested should be visible through the explorer (or other 3rd party tools). The explorer is totally open source, and we welcome contributions to improve it and display more/better statistics.

3. Will delegators be able to see all the services and locations registered by Orchestrators in service registry? Will that be available on livepeer explorer or in some other place?

Yes. The explorer should give a view into the registry, yes.

4. How do the protocol know whether the mistake happenned is from Broadcaster side (not paying amount properly) or from Orchestrator side (not performing work properly)? How do you ensure that Orchestrator is not getting slashed because of Boradcaster’s mistake?

An O can’t get slashed due to a B’s mistake. There is proof of the input segment signed by B, and only if O signs an encoded segments attesting to the output encoding, can O be slashed. If B doesn’t pay enough, O simply doesn’t do the work, there is no slashing in that case.

If B spends more money than they have, then their “penalty escrow” can be lost (which is kind of like slashing). B should be able to control for accidentally doing this with high probability if they have a big enough deposit in their penalty escrow.

5. What are some of the steps that delegators should take to ensure that they are delegating to the right orchestrator?

The same steps they’re currently taking - researching a combination of on chain statistics on past performance, the economics the O is sharing, and their offchain social campaign. It also isn’t unreasonable to “diversify” across multiple O’s to reduce risk and to encourage decentralization.

6. Centralization risk…

Certainly if the most efficient operators who can charge the lowest possible prices continue to provide great service, share the most fees, and attract the most stake, then they will likely win much of the work. Two factors which work against this:

• The more stake they attract, the more ways the fee-share is being split - hence a lower ETH/staked LPT fee ratio for delegators. If any up-and-coming node can attract work due to being available with good performance, a lower price, or more attractive fee share - then stake would likely shift over towards this node in favor of a higher ETH/LPT fee ratio.
• Local nodes have an advantage - if an up-and-coming node happens to be located closer to a B, then they’ll potentially have a lower latency response time on the transcoding, and hence they’ll win work, and attract an outsized ETH/LPT fee ration - attracting further delegation.

So yes if there are only 2 big B’s on the network, and 2 big O’s are serving them reliably as cheap as possible, there may be some centralization, but that’s a pretty small network…let’s onboard more diverse B’s, with different use cases, in different locations, building awesome video products

8. what is the approximate timeline for Streamflow to go live? 6 months? 1 Year?

The build has already been under way, so we’d like to have an internal testnet before the end of December as the next milestone. From there, of course external testing, reviews, audits, etc - so I can’t predict a date yet, but we’re working hard!

9. Do you have any broadcasters ready to start broadcasting on livepeer once Streamflow goes live? If yes, what is the volume (in terms of USD) that you are expecting?

Well, we’ve had a bunch of B’s use Livepeer already - but for one off events like conference and meetup streaming this isn’t exactly scaled transcoding. So using the term “Broadcaster” is actually a little bit of a misnomer, when in reality the target users are developers building scaled video applications. And yes, while there are a number of developers/projects who have expressed interest and would benefit tremendously from using LP post Streamflow, it is definitely on Livepeer to showcase that the network can be reliable, performant, and cost effective before they would turn on the integration. This is the purpose of Streamflow

10. Are there any transcoders in livepeer network that are currently earning fees (ETH) by doing transcoding job?

Yes, but low volume ETH - mostly through demonstrations, early integrations, and one off tests. One of the challenges that Streamflow addresses is that if the streams are short, then it actually costs more in ETH to do the txns to claim the fees, than they would make in fees - and hence not worth it. In Streamflow that’s no longer the case.

11. Is there any waiting time between broadcaster paying the fees, orchestrator receiving the fees and distributing that fees to delegators? Will that distribution be done at the end of each round or is it done at orchestrator’s discretion?

As soon as O receives a winning PM ticket (payment from B), they can cash it in on chain at their discretion. The default is that they’ll cash it in right away, and there’s an expiration date which means they can’t wait too long. This will automatically split the fees between O and its D’s - though the D’s will need to “claimRewardsAndFees()” for each round in order to move the fees into their wallet. This is a bit of an implementation/accounting detail though and could potentially be changed to work differently.

---

Hope that helped!

[velvetdoctor]

Awesome! Thanks a lot Doug. Appreciate it.

[dob]

Just wanted to add a note to this thread to say that I’ve updated a version of the paper to reflect much of the feedback to date, have published to master, and have written a summary blog post!

• View updated Streamflow Paper
• View summary blog post

[herve]

Is there an estimated rollout date/period for streamflow if all goes well?

[dob]

Target is to run an internal testnet prior to the holidays in December. From there it will likely still be a few months to get through finishing the implementation, testing, public testnet, audits, and rollout. If you want to follow along with progress the “Project” boards in github in the protocol repo and go-livepeer repo show what is being worked on - though it would also be great if someone wanted to organize a visualization of all collective Streamflow progress + to-do’s. Cosmos has a great roadmap visualization that could serve as inspiration.

[dob]

One question I’ve heard is: What are the downsides of Streamflow for current Transcoders?

• More competition - if there are hundreds of O’s instead of just 15 then it will be more competitive to win work and attract stake on the network.
• Necessity to actually compete to perform work on more than just quoted price - you’ll actually have to perform it fast enough at a low enough price, with high enough reliability to continue to perform future work. This may mean running better hardware, with more bandwidth, at access points with good connectivity.
• DevOps skills required to scale your setup. This isn’t necessarily new to Streamflow, but the O/T split means that you may want to run multiple T processes, monitor them, etc. Note: pre-streamflow would still have these sorts of challenges to scale.
• If you aren’t staking towards yourself, and you aren’t competing to win any work, and delegators move towards nodes who are winning work and sharing fees, there’s a chance that you would fall below the minimum active stake requirement and become deactivated. Note: min stake is only one option being considered for expansion of the network. Currently it keeps raising with a limit of 15 active nodes, but one option considered is to fix it an a known, accessible amount that provides enough security as a next step.

These are all things that may make your life as a Orchestrator more difficult than it is today in the pre-significant-work, pre-significant-fee, inflation only world. But I would argue they are all things that are good for the network’s reliability, performance, scalability, and decentralization. One more thing to keep in mind:

1. None of the above effects the status quo, where if you stake, you earn inflation. That continues whether you scale your setup to compete for work or not. You just may be better off delegating and earning someone else’s fee share, than running an idle node that misses out on its own fee share if not competing and winning work.

[haduong]

Thanks for the proposal, Doug & Livepeer Team!

I have only had the time to look at the various proposals so far without going into Economic Analysis, risk vectors etc but here are a couple of first thoughts:

• great summary of current challenges/weaknesses. Especially agree with 5 as I mentioned during the BlueYard session Also think 6 is important to further decentralize (which you haven’t mentioned in the Abstract regarding topics addressed by the updates)
• like the split up into orchestrator vs transcoder. More modular roles invite more participation and experimentation (those who still wish to continue working in the current modus can decide to vertically integrate and be both orchestrator and transcoder). I worry about the renaming though (current Transcoder -> Orchestrator) → might cause confusion
• there might be the potential for centralization for the new transcoders since their business model will have economies of scale (“transcoders with advantages in electricity, bandwidth, and geolocation will likely outperform those running without these competitive setups)
• Re “Relaxation of Transcoder Limit and Stake Enforced Security” - agree that this part is most open. I would lean towards option 2 or 4 but feel like 4 leads to a reputation-based system that benefits first movers & disincentivizes new Orchestrators to join
• not sure if excluding orchestators’ price is good since degelators want to know that competitive prices are offered (otherwise, delegating to that orchestrator won’t make much sense)
• like that location and service identifier are offered. There should also be a macro view that shows which locations & services are underprovided right now so orchestrators (and their delegators) can even out demand & supply
• “Offchain Job Negotiation” → I think this needs to be specified and analyzed more re potential added complexity/friction for different stakeholders or added risk vectors in comparison to the status quo. Imagine: a) Bob wants to outbid all other Orchestrators in price. b) Bob registers as a Broadcaster with a minimum job. c) He spams all Orchestrators in the network off-chain (writing to all of them to get quotes). d) Orchestrators will be incentivized to supply a quote as fast as possible so many will automatically just respond to every request. e) Bob aggregates the entire response data to find the market pricing distribution. f) Bob then uses that information to become an Orchestrator and offers better pricing than everyone else. g) Alice thinks the same as Bob, repeats all of these steps and underbids Bob. h) Eventually, the system will endure a price war & a race to the bottom while the amount of spammy off-chain communication is huge
• I feel like there is more responsibility on the Broadcaster’s side with this update (responsible for verifying the segments). Will that cause relatively more friction than now? I like the idea of outsourcing to a high quality Orchestrator (or Truebit). Would it be possible to offer that automated workflow in the interface of Livepeer directly to decrease complexity for new broadcasters?

PS: This comes from an email thread and we just wanted to share this publicly. Sorry for potential duplication and very casual wording

[dob]

Great feedback @haduong. Here’re some of my takes on the points you raise:

• there might be the potential for centralization for the new transcoders since their business model will have economies of scale (“transcoders with advantages in electricity, bandwidth, and geolocation will likely outperform those running without these competitive setups)

Yes, this would be the case if the location of the nodes didn’t matter at all - but with the latency requirements of video, nodes who happen to be located nearby the source videos/orchestrator/object stores will have an advantage in winning the performance race. Especially as you can imagine mobile clients capturing video across millions of devices all around the world, the network benefits from a geographically diverse footprint, which should create opportunities for localized competition. It’s not perfect against the economies of scale benefits that you mention, but it feels more naturally decentralized than, say, mining operations.

• not sure if excluding orchestators’ price is good since degelators want to know that competitive prices are offered (otherwise, delegating to that orchestrator won’t make much sense)

Can this transparency be provided by an offchain service? It could be gleaned from payment parameters on PM tickets most likely (which get cashed on chain), so a monitoring service could just report on the current state of pricing. Putting the price data on chain creates less flexibility because there’s a significant cost to updating pricing - and with swings in gas prices and crypto prices, this could add significant cost.

(with regards to offchain price quoting…) Eventually, the system will endure a price war & a race to the bottom while the amount of spammy off-chain communication is huge

Very good point, thanks for surfacing this. This is definitely something that deserves more work and research. I’d imagine nodes could use traditional DDoS prevention mechanisms to mitigate this a bit. More crypto-economically, you could deter this kind of attack by requiring a PM ticket to be sent when requesting a price quote - something small enough that it would be meaningless to an actual B, but would add up to be meaningful in a sybil/spam attack.

• I feel like there is more responsibility on the Broadcaster’s side with this update (responsible for verifying the segments). Will that cause relatively more friction than now? I like the idea of outsourcing to a high quality Orchestrator (or Truebit). Would it be possible to offer that automated workflow in the interface of Livepeer directly to decrease complexity for new broadcasters?

Yes, the Livepeer node will automatically handle almost everything in Streamflow for a B. The idea is to off the B’s “strategies” for each decision point - O selection, redundancy, performance requirements, verification, pricing. There would be sensible default strategies that would just work for the average use case, but then a developer could certainly replace the default strategies with other implementations or custom strategy. I’d imagine an open source strategy repository where people could pull down what they needed. Actors using different strategies actually create a more unpredictable, harder to game, network - whereas a single implementation that works the same for every user would be much easier to game.

It’s worth remembering that the B’s, are really developers implementing applications, rather than individual streamers. So think of the developer behind application X making a decision about their reliability requirements up front, and possibly adjusting it over time, as opposed to each user of said application actually needing to think about any of this.

Thanks again!