Following the successful completion of the AI Metrics and Visibility proposal, the Livepeer Cloud SPE has continued to advance the network’s performance monitoring infrastructure — even beyond the funded scope — to ensure that Livepeer remains the most reliable, transparent, and innovation-driven decentralized video compute network.
Background
Our original proposal delivered:
- The AI Job Tester — a robust benchmarking service executing inference workloads against active AI Orchestrators.
- Enhancements to the Leaderboard Data API, supporting both AI and transcoding job data.
- Integration with the Livepeer Explorer, providing visibility into orchestrator performance through a dedicated AI Performance Leaderboard.
These features collectively laid the foundation for standardized measurement of AI job performance across the network.
Recent Work: Bringing Live Video Testing Online
Since completing that proposal, the Cloud SPE team identified and addressed a critical gap: support for live video pipelines.
To bridge this, we undertook significant new development — without additional funding — to extend the tester’s capabilities and gain early visibility into live AI video job performance.
Key accomplishments
-
Live Stream Input Support (RTMP/FFmpeg)
The AI Job Tester now supports pushing live video streams into the network and validating playback of the AI-enhanced output through the Livepeer Gateway and MediaMTX. -
New Scoring Logic for Live Video Jobs
Previously unscored job types — such as real-time AI video processing pipelines — now have purpose-built scoring routines, providing the first generation of live video performance data. -
Integration with MediaMTX and Multi-URI Management
Because the on-chain service registry does not yet support multiple service URIs or discovery for Daydream workloads, we implemented off-chain logic to map and reconcile service locations for accurate testing and reporting.
These efforts have resulted in live video job scores appearing in the Livepeer Explorer, representing an early but meaningful milestone toward comprehensive real-time metrics for the network.
Challenges & Lessons Learned
During this process, several network-level insights emerged:
-
Service Registry Limitations
The current registry lacks full support for live AI jobs, requiring hybrid on/off-chain discovery methods to identify and test available services. -
GPU Saturation
Many orchestrators are fully utilized with live workloads, making it difficult for the tester to acquire capacity for consistent, repeatable scoring. -
Rolling Window Scoring Variability
With scores based on 24-hour rolling windows, intermittent GPU availability causes fluctuations that obscure true performance trends. -
Need for Smarter Sampling & Verification
We evaluated job-availability detection and realized that deeper network-wide activity verification will be critical to prevent gaming and ensure fairness in future scoring models.
What’s Next
While the latest work introduced new complexities, it also provided the most comprehensive understanding yet of what it will take to deliver state-of-the-art measurement and visibility across both AI and live video workloads.
In the coming weeks, the Cloud SPE will publish a follow-up outlining a forward plan to:
- Improve reliability and fairness in testing,
- Enhance data collection and orchestration transparency, and
- Evolve the Livepeer Explorer’s metrics to reflect real-world live AI usage.
Closing Thoughts
The Cloud SPE continues to operate the AI Tester and advance the Livepeer metrics ecosystem — without any additional treasury funding — because we believe deeply in the mission of building a more observable, performant, and trustworthy network.
Our progress demonstrates not only technical capability but also a commitment to long-term collaboration and accountability. We look forward to engaging with the community on how these learnings can inform the next phase of AI and live video metrics on Livepeer.
As always, we welcome your feedback and discussion below.
— Livepeer Cloud SPE
Mike Zupper (@xodeapp.xyz*) • Speedy Bird (@speedybird.eth)
Github Repo: GitHub - mikezupper/livepeer-ai-job-tester at tasks/live-video-support
Links to previous work on this topic:*