Proposal for a Fairer and More Decentralized Stream Distribution System

Governance
1

Dear Livepeer community,

I want to bring to your attention an issue with the current stream distribution system on Livepeer. As it stands, the system doesn’t promote decentralization and equal stake distribution among orchestrators. Many delegators tend to stake on orchestrators that already hold the most LPT, making it difficult for newer orchestrators to gain a foothold and establish a diverse network of contributors.

Furthermore, the weight of LPT stake in the selection process is too high. This forces broadcasters to distribute their work randomly among orchestrators, in order to mitigate this issue. This isn’t ideal for either broadcasters or delegators, as it results in inefficiencies and an overall suboptimal user experience.

Therefore, I propose two solutions. Firstly, the weight of LPT stake in the selection of orchestrators should be limited to a 1:10 ratio, possibly through implementing a quadratic system. This would incentivize delegators to distribute their stake more evenly among orchestrators, leading to better returns and a fairer distribution of streams among them.

Secondly, we should consider implementing a maximum saturation level for nodes, beyond which staked LPT wouldn’t be taken into account in the selection process. This would be similar to the approach taken for staking nodes in the Cardano network. This would encourage a more diverse and decentralized network, which would benefit the Livepeer ecosystem as a whole.

An example of the problem is that the biggest orchestrator right now has a 99% chance of receiving a stream vs an O that has 20,000LPT. This basically means the only way an O can win a stream is through the 30% random logic. This does not make sense. There are small staked orchestrators with advanced and very performant architecture that are getting no streams and earning no return on their investment.

Regardless of what solution the community comes to, It’s time to address these long-standing issues with the stream distribution system that were supposed to be eased with the Explorer update but frankly have just gotten worse.

I welcome your feedback and hope that we can work together to create a more equitable and efficient stream attribution system.

@dob @yondon

Best regards

11 Likes
2

Yes please, I’ve been wishing for a chance like this since I started.

4 Likes
3

This topic keeps resurfacing time and time again. We have discussed it multiple times, but it seems to always fall by the wayside. Due to the distribution of jobs, multiple new orchestrators have left the network because it is challenging to keep up with larger orchestrators. This difficulty is primarily due to the weight of stake in the selection algorithm. Smaller orchestrators only receive jobs through a 30% random distribution of work, rather than being selected based on their performance. This situation seems entirely unfair.

Would it be possible to make some changes to the selection algorithm? For example, we propose making it performance-based up to a certain real-time ratio or imposing a limit on the weight of stake in the selection process capped at a certain amount.

4 Likes
4

as Pon said, this subject resurfaces regularly, and rightly so.

It appears that despite the efforts of the core team to encourage the delegators to better distribute the stakes, and despite the efforts of the smaller orchestrators to attract more stakes, it is impossible for them to compete with the larger orchestrators.

Solutions as proposed here could certainly allow a better distribution of work, a better decentralization, and avoid having to use temporary solutions like distributing 30% of the work randomly to mitigate this problem.

This subject deserves at least a discussion to try to see if acceptable solutions exist to solve this problem.

4 Likes
5

Whole heartedly agree on this. I think we can safely say that development on go-livepeer has stalled a bit since the Arbitrum migration. Besides making us ready to meet market demands of the near future (AV1) and a more advanced pricing model (price, speed, quality), one of the biggest pain points is job distribution.

We got a whole lot of orchestrators with a reliable, professional setup looking at Vires’ dashboard with envy. They only get a fraction of the streams even though they are just as involved (if not more) with the Livepeer ecosystem in terms of being part of the community, writing guides or sharing their tools.

In order for them to make being part of Livepeer worth it, they want a return on the time and effort they put in. I think it is troubling to see new orchestrators come in highly motivated initially, but then quietly drop into ‘maintenance mode’ or even move on alltogether because in the end, it’s just not whay they expected from running a node.

The grant program alleviates this a bit, but is cumbersome and relies on Livepeer Inc to judge applicants and dish out rewards. The latest explorer changes to show ROI help out very small orchestrators who are just starting out, but since ETH fees are still a drop in the bucket it still comes down to lowering your reward to cut as much as possible, especially if you want to have a chance of one of the few whale delegators staking towards your node.

Even though ETH fees are too little to make a difference to delegators, they can be significant for orchestrators. Lowering the reward cut to 4% or lower to be competitive is just not worth it, unless they would be able to offset this with transcoding fees.


I think OP’s proposal of ensuring there are diminishing returns on having more and more stake makes a lot of sense, but I think we need to take this a step further by having broadcaster nodes collect and consider more (off-chain) performance metrics.

We need to keep working on this until nodes like Binance, who offer no value and have never performed well stop out-earning orchestrators who actually enable this decentralised transcoding network

6 Likes
6

I fully agree on changes, as it’s very difficult to compete regardless how good your set up. I think both proposed solutions should be implemented to tackle the issue. It would make network more decentralised and fairer on smaller orchestrators.

3 Likes
7

I’m generally in favor of the intent of this proposal - but to make this actionable, we need to really dive into the specifics and model out different scenarios.

What we’re talking about here is a change to the selection algorithm of Livepeer Inc’s implementation of the Livepeer protocol. This would not affect other implementations such as Xeenon’s, as they use bespoke logic. It would also not impact any future client implementations.

I am a little nervous about implementing inflexible models - an inflexible model has created the scenario you describe, and I think the option to “turn the dial” is important here. One cool solution to this problem is TheGraph’s use of a Cobb-Douglas function to distribute fees [1].

I am very much in favor of tweaks to fee distribution. I am currently against a saturation threshold (change my view!) because fee distribution updates seem like a more elegant solution to the same problem.

[1] The Cobb-Douglas Function & the Math Behind Web3 - Edge & Node Blog

6 Likes
8

as a small orchestrator I strongly agree with the notion for a more fairer and more decentralized stream distribution system as it would encourage orah’s like me to keep going.

3 Likes
9

I hope one of the giga-brains in the community gives you a well-thought-out response. bit above my expertise.

All I can say is I hope we find a solution soon, even if it’s a temporary band-aid fix. The fee/stream system is in dire need of change. I can’t stress how much effort Orchs are putting into their operations, some of us going as far as to build complex pools on the network. For me personally, running Livepeer nodes has become an almost full-time job. I want to be able to count on my earnings to support further scaling, expanding, and onboarding efforts (including onboarding investors).

3 Likes
10

This problem represents a great opportunity for the tech-savvier community members to join heads. We all want to solve this problem and we have the know-how to do it. Would anyone be interested in starting a working group for this issue?

3 Likes
11

I don’t get what you mean by “limited to 1:10 through a quadratic system”, can you elaborate? How many streams should an Orch with 2M stake get vs. an Orch with 20k stake according to you?

Also your proposed “solutions” simply won’t work in a decentralized network. If you were to limit the max stake to e.g. 100k, bigger nodes would just start spinning up a 2nd, 3rd, 4th etc. node. All running on the same hardware, so no additional cost. The only thing you’d achieve is setting the stake requirement to enter the top 100 a lot higher.

So it’s not really 99% then, is it? :unamused:
If you look at the public dashboards and compare the amount of streams vs. the amount of stake, you’ll see that it already heavily favours smaller stake. E.g. Authority Null has 10x less stake but only 4x less streams than me. I also have a testing node with 20k stake, it’s on the same hardware as my main node: 100x less stake, 25x less streams.

Since we only really have one broadcaster, this would make the network a lot more centralized.

In general:
A lot of the reasoning seems to be “decentralization”. So just to be clear (since some are making comparisons to L1s like Cardano): Livepeer’s security and functionality is NOT dependent on having highly diversified Orchestrators. Currently, the top 20 Os by stake could all go down and it probably wouldn’t even matter to the workings of Livepeer. If you all care so much about decentralization, we should probably rather talk about the 2 of 3 multisig that secures the whole network :wink:

Also, even if we were to equally distribute the amount of work among all current, good performing nodes, it would probably result in 1 or 2 more streams per node. I’m not convinced that this would make a difference to the smaller nodes. And that’s only in the short run. If one can get a decent revenue from streams with only 500 LPT staked, new Orchs will enter and therefore the streams per O will go down again.

Instead, what we should really focus on: How do we stop paying thousands of LPT each round to nodes that don’t even transcode? If we can somehow measure this in a decentralized way and redirect those LPTs to nodes that contribute to the network AND to attract new broadcasters, it would IMO have a much bigger effect than getting a stream or two more for a short period of time.

3 Likes
12

@vires-in-numeris All your points are valid, and i’m not smart enough to challenge them on a technical level, but do you think the fee distribution we have is fair at the moment?

streamdistribution
streamdistribution833×2950 288 KB

2 Likes
13

Oh come on… posting a highly selective screenshot based on a few hours…

Let’s look at the last 90d on the explorer:

image
image861×554 59.5 KB

Now divide the fees by the amount of stake and tell me in what way the current work distribution is unfair to smaller nodes?

2 Likes
14

Ok, but why is it 1 month my node earns 2-4 tickets, or even the rare 0, and how do you propose we close the gap on stake organically?

I think the trailing 90d fees is a poor representation of just how inconsistent winning tickets are for smaller nodes.

Why is Xodeapp #9? He has one of the most performant setups in the entire network.

Why am I #3? What’s the logic behind that?

Why did I crush it last month but barely see tickets this month? Why are you making ~$450 some days (before fees), when you weren’t making nearly as much in the months prior?

1 Like
15

And just to add to this, from your perspective, do you not see an issue with the fact that the stake gap between your node and any of the other high-performing nodes is around 1.6M LPT?

1 Like
16

Work is volatile and luck based - e.g. I recently won a 0.18 ETH ticket during the ARB launch when gas prices were crazy. That’s exactly why you need to look at longer intervals…

It’s not an issue for the network functionality/security. If I go down, we have more than enough nodes that will take over.
That said, it sucks that it’s difficult to attract enough stake for newer, high performing nodes (even if they obviously put in a lot of work and support the network/community with great tools/improvements). But changing the work distribution so that my streams are halved and everybody else gets one more stream will hardly fix that…

A mid term solution could be to stream some LPT each round to the grants program. And reward high performing and contributing nodes through that. Until we can measure “performance” in a decentralized way or find a better solution.

3 Likes
17

Sorry for the spam. I tend to rapid-fire my thoughts. I also recognize my tone on this issue can seem aggressive - not my intention, but I am passionate about it.

I do think there is something to be said and eventually addressed regarding the stake gap. However, in order to purpose a real, actionable solution, we need to understand the problem better. Only then can we really move forward with anything…

Work is volatile and luck based - e.g. I recently won a 0.18 ETH ticket during the ARB launch when gas prices were crazy. That’s exactly why you need to look at longer intervals…

Very true, but there have been months where I’ve averaged a high stream count, even with high-value streams, and seen fewer winning tickets than months with lower stream counts and lower-value streams. This tells me something is actually broken with the PM system.

There are clearly 2 issues trying to be tackled here… “is the stake gap a problem?” and “what the heck is up with winning tickets?” I hope we can come together as community and find concrete solutions.

2 Likes
18

Looks like a lot of energy around this discussion, which is great. A couple things to note…

  1. It’s worth pointing out that stake, as theoretically representative of security, is only one input into the selection algorithm. Performance, location, work history, and randomness are others. (acknowledging that fast+full verification and slashing would need to be activated for the stake-as-security impact to fully be realized). It is certainly possible to fork a client implementation and reconfigure the client-side selection algorithm to take this into account differently and rebalance the algorithm. Even better, you could make those parameters configurable or swap out different selection algorithms via a startup parameter. This might be a nice bounty or grant.

There’s a second challenge where the majority of video flows through one large broadcaster, so Livepeer Studio’s B would have to be willing to experiment with running these different clients and observe the impact on performance.

  1. Would the juice be worth the squeeze at the current moment? Completely ballparking here, but if people are concerned about the top staked node earning ~6 ETH over the last 90 days, versus the next 10 nodes in the 1-2.75 ETH range, if you assumed a more “fair” algorithm would lead the top node to only earn 3 ETH, then this would lead to another 0.1 ETH for each of the next 30 nodes if they were the beneficiaries…or even less 0.03 ETH if split amongst all of the nodes. Would this difference really make node operators feel more fairly rewarded for their effort?

Or is distribution of LPT from the protocol the bigger opportunity to tackle? On a separate note I’m thinking that we need to better fund public goods via protocol inflation (both demand side and supply side O level contributions beyond just node operation), and that could be a good opportunity to create mechanisms to better route protocol issued LPT into funding pools. Topic for another thread.

5 Likes
19

While the trailing 90d fees wouldn’t change too much, it may provide a more consistent and predictable flow of fees for orchestrators who currently cannot find any pattern in earned fees. Also, a more aggressive focus on performance for selection would benefit the network as a whole and spark some really creative innovation.

I’ll be blunt here. At the moment, high-performing orchestrators can compete against all other high-performing node operators except Vires and that doesn’t seem healthy.

The distribution of delegated stake is a big issue, and while many of us have tried to find creative ways to close the gap, it only seems to be increasing.

What I’m trying to understand is why winning tickets are so inconsistent, even when streams are consistent. What changed this month that caused the payout bot to light up green?

2 Likes
20

If you can tell me how we can measure performance in a decentralized way that can’t be gamed, I’m all for it. As long as the performance is defined by one entity and with gameable parameters, it would hurt the network and not benefit it.

I’ve been an active community member since 5 years and have been running an Orch since 4 years - without missing a reward call and while having a consistently low reward cut. That probably gives me quite a unique branding as a highly reliable (“stake and forget”) Orch, especially for larger stakes. E.g. 5 stakers account for over 1M.
When I started, there was obviously less competition. But it still took me a while (and contributions like the telegram bot) to grow… and 4 years (5 if you count project involvement) to get where I’m now.

Different stream formats combined with the variance of micropayments? Also your pixel price is 900 and mine is 1200, so that could explain some things?

But I’d appreciate if we could stop making this about me and instead focus on the bigger picture

2 Likes