A retrospective on the last six months of work by the AI SPE per our Phase 4 Livepeer Treasury Proposal
Note: The AI SPE is now restructured under the name MuxionLabs.
Introduction
Phase 4 built on the Phase 3 real-time foundations by hardening Bring-Your-Own-Container (BYOC) workflows, expanding ComfyStream’s capabilities (including text/data-channel outputs and multimodal support), and shipping applied-research additions to the ComfyUI Stream Pack. The focus was to move from “prototype” to “repeatable experimentation” while unblocking a path to broader orchestrator and demand onboarding.
Significant progress was made toward productionizing the BYOC pipeline. BYOC streaming has been merged into go-livepeer with substantially reduced coupling to the existing ai-runner path and improved test coverage. Earlier work that did not made it into the main branch still served a valuable purpose by validating the core architecture and enabled testing sustained streaming workloads on the network.
During this process, earlier assumptions about directly embedding BYOC streaming into existing go-livepeer execution paths proved limiting. Addressing this required a deeper refactor than initially planned, resulting in a temporary revert and subsequent redesign. The outcome was a fully decoupled BYOC execution path and the emergence of PyTrickle as a standalone integration layer, enabling containerized Python workloads to participate in Livepeer streaming with minimal friction.
This tradeoff delayed some downstream milestones but materially strengthened the long-term viability of BYOC as a generalized, permissionless onboarding mechanism.
By the end of the phase, we materially stabilized the BYOC path in go-livepeer and made it a clearer, repeatable route for teams to bring new job types onto the network - this has unlocked the ability to onboard multiple demand sources without bespoke core integrations.
On the demand side, the Embody SPE has been pushing production workloads to the Livepeer Network via our BYOC solution for the past six months, recently transitioning to a custom implementation. The Streamplace integration is now complete and expected to go live in production shortly. In parallel, we’ve published multiple example applications at
showcasing fully end-to-end pipelines on Livepeer.
The AI SPE is in the process of rebranding as MuxionLabs.org, but this retrospective will refer to the original name as when we delivered the Phase 4 proposal.
Commitments Delivered
These Deliverables reflect the most meaningful progress made in Phase 4:
Stabilized BYOC batch processing
BYOC matured through hands-on iteration with the Embody SPE, BYOC batch-style request bugs were fixed and the batch path is now largely stable. This enables repeatable batch BYOC jobs on the open network.
BYOC streaming in production
BYOC streaming was implemented early in Phase 4 and successfully demonstrated sustained operation on the open network, running continuously for multiple hours. While the initial implementation was temporarily reverted to allow for additional review and test hardening, this work validated the core streaming architecture and clarified the remaining integration requirements with the core network stack.
Following this, we expanded test coverage, improved stability, fully decoupled the BYOC path into a standalone module, and landed the updated implementation in the main branch of go-livepeer. Ahead of the official merge, the Embody SPE has been actively leveraging this work via beta Docker images built from our go-livepeer fork.
ComfyStream Optimization and Expansion : text output + multimodal + BYOC
ComfyStream gained text output support for nodes such as audio transcription that produce data as opposed to images/video. Additional enhancements like dynamic warmup and automatic modality detection reduced operational friction and increased the range of workflows a single worker can serve.
We enabled ComfyStream to run on the Livepeer BYOC stack. Now ComfyStream can serve not only as a fast prototyping tool for new real-time video AI pipelines by developing workflows in ComfyUI, but also a robust solution for hosting multiple models and disparate workflow/pipelines on one orchestrator in a single container. Quickly warming up a new pipeline and models on demand - even mid stream - is a built in feature.
Stream Pack improvements and Real-Time Video Custom Nodes
4 sets of Custom Nodes were added to the ComfyUI Stream Pack.
-
StreamDiffusion + SDXL + TensorRT + Loras + Controlnets + IP-Adapters
- Ported from Livepeer INCs Daydream StreamDiffusion pipeline
-
StreamDiffusionV2
-
Ported from Livepeer INCs Daydream StableDiffusion 2 pipeline
-
Supports V2V & I2I
-
-
SuperResolution
- Real-time video upscaling
-
AudioTranscription + SRT
-
Synchronizes captions into video output
-
Alternately can output transcribed text only via dedicated data output custom node
-
The phase also produced four real-time ComfyStream pipeline/workflows accompanying these node sets, to enable creative generative real-time streaming video, audio transcription to a data channel (Whisper with text output), and transcription with SRT subtitles embedded in the output video segments.
In addition, we have implemented a simple, config based method to allow for developing and deploying workflows using custom nodes which have different underlying python package requirements. ComfyStream docker/dev containers can now be built with a user specified subset of supported Custom Nodes from the ComfyUI Stream Pack.
Delivered Beyond What Was Committed
Though these items were not explicitly specified in the scope of Phase 4, we determined that these strategically important force multipliers expand on the work that we have done on BYOC and ComfyStream in the spirit of our mission to make it easy to build on the Livepeer network.
Alternative worker integration path via PyTrickle
The AI SPE developed the PyTrickle BYOC package, which allows Python applications to operate as Livepeer AI workers outside the ai-runner integration path (livepeer/pytrickle). PyTrickle provides a modular BYOC serving model, enabling Python codebases to participate in the Livepeer network by importing a single package. Pytrickle docs can be found at https://muxionlabs.github.io/pytrickle/
We’ve also integrated PyTrickle into ComfyStream, demonstrating how an existing application can quickly adopt the BYOC stack while delivering a flexible, production-grade backend AI worker.
Developer Tooling for BYOC Streaming Experimentation
To lower the barrier for builders and orchestrators to develop and test real-time BYOC streaming pipelines, the AI SPE shipped example-apps.muxionlabs.org. This site showcases several ready-to-fork, BYOC-compatible open-source applications that demonstrate end-to-end streaming workflows on the Livepeer network. The underlying implementations live in the BYOC Example Apps GitHub repository, which also includes a lightweight web UI for testing PyTrickle- and BYOC-based pipelines.
In parallel, we published a TypeScript SDK for BYOC streaming (@muxionlabs/byoc-sdk) to simplify client-side integration. The SDK enables front-end applications to stream directly from the browser to a Livepeer Gateway without requiring a custom backend. It provides WebRTC streaming, data-channel support, device management, React hooks, and connection utilities such as retries, monitoring, and statistics.
Together, these tools reduce integration overhead and make it significantly easier to move from experimentation to deployable prototypes. They are already being used in practice by the Cloud SPE for validation and by orchestrators for debugging and pipeline testing.
ComfyUI Daydream API node
Recognizing an opportunity to translate emerging ecosystem demand into scalable, extensible infrastructure, the AI SPE developed the ComfyUI-RTC custom node set that enables real-time webRTC streaming workflows directly within ComfyUI, allowing users to connect live frame streams to external APIs and backend pipelines.
While Daydream served as an initial design partner, the node was intentionally built as a brand-agnostic integration layer that can support any compatible real-time generative or streaming backend.
The node includes a local RTC server that streams video frames to external services and supports bidirectional workflows, including extracting the latest frame from an active stream for downstream processing within ComfyUI. This architecture enables flexible composition of real-time pipelines without requiring a local GPU, across platforms.
By partnering with Livepeer Inc. and validating the integration through Daydream, this work identified a clear demand vector for real-time streaming within the ComfyUI ecosystem. At the same time, the generalized design positions the node as a reusable foundation for future integrations, enabling additional demand spikes and third-party adoption across the ComfyUI community.
Milestones (Retrospective Status)
Phase 4 was primarily focused on two objectives outlined in the proposal:
(1) maturing BYOC from an early prototype into a production-ready, permissionless pipeline, and
(2) expanding ComfyStream and the ComfyUI Stream Pack to broaden real-time AI demand on the Livepeer network.
While not all milestones were delivered exactly as originally scoped, the phase produced meaningful architectural progress, validated key assumptions, and resulted in durable infrastructure now landing in core Livepeer components.
Demand Partner Onboarding and Validation
The Agent SPE and Embody SPE served as primary early design partners throughout Phase 4. Extensive collaboration with these teams drove critical improvements to the BYOC stack, particularly around stability, execution semantics, and streaming behavior.
While the proposal targeted onboarding three additional demand partners, progress here was more incremental than planned. Work was initiated with Streamplace, including an MVP transcription pipeline and coordination with the repository owner; however, full onboarding was gated on additional go-livepeer changes currently under review. This experience reinforced the need to harden the core BYOC path before scaling partner onboarding.
Technical RFC and Public Roadmap
While BYOC RFC content was discussed publicly through watercoolers and ongoing conversations, a formal published RFC was not released during this phase. The PyTrickle library emerged as a foundational component of the BYOC stack, serving as the integration layer between Python-based, containerized applications and the Livepeer real-time streaming stack.
A comprehensive Technical RFC was completed in the form of the BYOC PyTrickle Engineering Spec for Custom Containers, defining the end-to-end architecture and serving as the canonical reference for BYOC development.
Similarly, while a public BYOC roadmap was discussed informally with stakeholders and partners, it was not released as a formal, published roadmap. This remains an actionable follow-up now unblocked by the stabilized architecture.
Orchestrator Onboarding
End-to-end validation of a fully production-ready BYOC pipeline on public orchestrators remains partially outstanding. While over a dozen orchestrators have been BYOC-backed workloads through collaboration with the Embody SPE and exercised via beta Docker images, broader onboarding was constrained by the earlier need to redesign BYOC’s integration with go-livepeer.
With the decoupled BYOC path now landed in the main branch the remaining work is primarily operational: merging final changes, deploying to orchestrators, and completing end-to-end validation on the public network.
ComfyStream and Stream Pack Expansion
Progress against ComfyStream and Stream Pack related milestones was strong and largely aligned with the objectives outlined in the Phase 4 proposal, particularly around expanding the diversity of real-time models, pipelines, and data-driven workflows available on the network.
Foundational real-time video models.
Three new real-time foundational video model nodes were delivered via the ComfyUI Stream Pack, meeting the proposal target:
-
StreamDiffusion
-
StreamDiffusion V2
-
Super Resolution
These nodes enable the community to build more diverse and higher-quality real-time creative workflows on Livepeer, and serve as reusable building blocks for both research-driven experimentation and partner-facing pipelines.
ComfyStream-enabled pipelines.
Five new ComfyStream-enabled pipelines were delivered, satisfying the milestone for demonstrating compelling, end-to-end workflows:
-
StreamDiffusion
-
StreamDiffusion V2
-
Real-time audio transcription
-
Transcript output via data-channel
-
Embedded Video Subtitles (SRT)
-
-
Video understanding via Gemma (BYOC container pipeline)
-
Generative video via a BYOC container pipeline
These pipelines demonstrate a range of capabilities spanning generative video, video intelligence, and hybrid video-plus-data workloads, and are intended to be reused by the ComfyStream creator community, internal research efforts, and external demand partners.
Data-channel–enabled models.
One data-channel–enabled model (real-time audio transcription) was delivered during the phase. While the proposal targeted three such models, early implementation work revealed that embedding data-channel support directly into the existing go-livepeer live-video-to-video path was overly restrictive. This led to a shift toward implementing data-channel support through the PyTrickle and BYOC architecture, resulting in a more flexible and extensible foundation for future data-channel–driven models.
Deferred Scope for Future Phases
Two items were intentionally deferred to future phases: publishing the BYOC Technical RFC and public roadmap, and scaling orchestrator and demand partner onboarding. Both were delayed to prioritize stabilizing and decoupling the BYOC streaming path in go-livepeer, ensuring the architecture is robust, reviewable, and suitable for long-term upstream support. With this foundation now in place, these efforts are unblocked and can proceed with greater clarity and confidence in subsequent phases.
Network Impact
Phase 4 strengthened Livepeer’s position as a viable platform for custom real-time AI workloads by materially reducing the engineering friction required for demand partners to bring their solutions on-network. By stabilizing the BYOC execution path and validating streaming use cases, we moved the ecosystem from bespoke core integrations toward a repeatable, modular onboarding flow.
This progress enables real world demand today:
-
The Embody SPE is actively running BYOC jobs, demonstrating that third-party workloads can be deployed and exercised on the network using the stabilized BYOC path and beta tooling.
-
The Streamplace transcription integration is in active staging and expected to go live soon, illustrating how external builders can extend Livepeer with new use cases without custom internal forks.
-
Multiple general reference pipelines (e.g.: generative video, video understanding, real-time transcription) published in the livepeer-app-pipelines repository provide concrete, runnable examples that other teams can fork and adapt without deep network expertise.
These outcomes matter because they convert architectural advancement into deployable demand use cases via scripts that builders and orchestrators can plug into and expand. While broader production hardening (orchestration stability, observability/metrics, SLA tooling) remains in flight, the engineering foundation now supports meaningful, concrete demand activity and makes future partner onboarding significantly easier.
Phase 4 didn’t just ship features, it unlocked the next wave of demand experimentation on Livepeer. Future phases will now focus on scaling that demand into reliable, open, fee-generating activity across more orchestrators and use cases.
Key Learnings
-
Shipping BYOC streaming required architectural maturity beyond ai-runner’s current state
Although BYOC streaming was technically viable early in the phase, attempting to integrate it too tightly into existing go-livepeer execution paths revealed shortcomings in testability, reviewability, and long-term maintainability. Decoupling the BYOC path and strengthening test coverage ultimately proved necessary to make the work acceptable for upstream inclusion and future iteration. This reinforced that production readiness on Livepeer depends as much on integration quality and operational clarity as on raw functionality.
-
Data-channel support meaningfully expanded the set of viable demand use cases
Adding first-class support for text and structured data alongside video unlocked concrete new workflows, most notably real-time transcription. These capabilities are already being exercised through ComfyStream pipelines and early partner integrations, and they broaden Livepeer’s applicability beyond purely generative video into video intelligence, analysis, and agent-driven use cases. This validated data channels as a key lever for diversifying network demand.
-
Hands-on usage by partners and operators surfaced the need for better developer tooling
Feedback from the Agent SPE, Cloud SPE, and orchestrators testing BYOC streaming highlighted that even well-designed APIs are insufficient without ergonomic tooling for debugging, iteration, and client-side integration. This directly motivated the creation of example BYOC applications and the TypeScript streaming SDK, which are now actively used for testing and validation. Reducing this friction is critical to moving from isolated prototypes to repeatable ecosystem experimentation.
Conclusion & What’s Next
Although several milestones were delivered later or differently than originally scoped, Phase 4 achieved its core objective: transforming BYOC from a promising prototype into a robust, extensible foundation for permissionless AI workload onboarding on Livepeer. The architectural decisions made during this phase — particularly around decoupling, modularization, and reuse — directly support the long-term strategy outlined in the proposal, even where short-term delivery targets shifted.
Using our newly shipped SDKs, new streaming BYOC streaming path through go-livepeer, and our proven pipelines, we have unlocked the ability to support the team’s goal of growing demand by making deployment easier, more reliable, and performant. Builders can now develop a deployable application on the Livepeer network either by using ComfyStream as a flexible real-time streaming backend, or by utilizing PyTrickle in their own container and the go-livepeer endpoints exposed by BYOC.