vllm-realtime: realtime speech transcription over Trickle + WebSocket

Retroactive Grant Applications
1

What did you build?

One sentence. What is the output?

A vllm-realtime example for livepeer/app-examples (issue #2): realtime speech transcription on the Livepeer network — PCM audio streamed in over Trickle, live transcript streamed back word-by-word over an orchestrator-proxied WebSocket, with live settings updates mid-stream (e.g. language) and full performance stats on both transports.

Why does it matter?

One to two sentences. What problem does it solve, and for whom on the network?


It demonstrates a non-video, non-droppable realtime workload (audio) running end-to-end on the network — the first transcription example — and ships a GPU-free mock backend so the whole Trickle/WebSocket pipeline runs on a laptop, plus a real vLLM/Voxtral backend for actual inference.

Link to the work

Direct link to the merged PR, deployed tool, published docs, or equivalent. No link = no review.


PR: https://github.com/livepeer/app-examples/pull/28 Issue: livepeer/app-examples #2 Upstream bug report: (the go-livepeer issue — link once you post it) FEEDBACK.md in the PR (developer-experience notes)

Evidence of impact

How is this being used? Who benefits? Quantify where possible (e.g. number of users, PRs merged, integrations adopted).


Verified end-to-end on an RTX 4090 with Voxtral-Mini-4B-Realtime: ~4× realtime throughput, 0.36s finalize tail, 14/14 segments with zero loss. While building it I found a silent data-loss bug in Trickle — subscribers join at the live edge and unread segments are destroyed with seq_gap_events=0 and no error — reproduced it (1 of 14 segments delivered), and fixed it in-app with backpressure. Reported upstream so the team's engineer can review.

Community proof points

Link to the Discord thread or other signal. Reminder: you should also get 2-3 Orchestrators commenting their support for this work.


Discussed throughout with Gideon (Livepeer) in Discord; the Trickle finding is being filed as a go-livepeer issue for the core engineering team.

Amount requested

USD-equivalent (max $5,000). Include a brief breakdown if the amount is above $2,000.


$500 USD in LPT Payout address (Arbitrum One): 0x813Cf34629207d1411209e4b9dAdA985A4f675Ad
Proposal: Network Engineering SPE
2

Decision: Approved

The Review Team approves the requested $500 in full, subject to PR #28 completing the normal merge process.

The contribution addresses the network engineering need established in Issue #2 by providing a self-contained realtime speech-transcription example using Trickle audio input, WebSocket transcript output, a GPU-free mock backend, and a real vLLM/Voxtral backend. It demonstrates streaming transcripts, live settings transport, lossless backpressure, detailed performance statistics, and both offchain and paid execution.

Independent review deployed the example through a registered Arbitrum mainnet orchestrator using go-livepeer v0.9.0 and publicly trusted TLS. Testing returned the correct transcript, delivered 14/14 segments without loss or retries, completed with a 0.437-second finalization tail, and recorded a paid PymtHouse session costing $0.005002.

The requested amount is modest and proportionate to the scope, technical complexity, network relevance, and useful developer-experience findings.

Before payment, PR #28 should be updated against the current main branch, its non-mergeable state resolved, and the normal CI, review, and merge process completed. It should also be updated to the current Livepeer gateway API; independent testing required a compatibility build because the submitted code targets an older mutable SDK branch.

The submitted client disables TLS certificate and hostname verification. Trusted certificate verification should be enabled by default, with any insecure mode restricted to an explicit local-development option.

The paid client currently makes only the reservation payment and discards the payment-session object. It should retain that object, send continuing payments during long-running streams, close it correctly, and include a paid-session regression test.

The live-language claim should either be demonstrated using a setting supported by the selected vLLM version or described as settings-transport functionality. The current implementation forwards the language field, but vLLM 0.24 ignores it.

The example currently assumes a locally hosted, unauthenticated remote signer. Its --signer option does not expose authentication headers or token refresh for hosted signers such as PymtHouse. This limitation does not justify reducing the award because the documented local-signer path exists and paid execution was independently verified.

Future Follow-Ups

  • Pin the Livepeer gateway commit, vLLM image, model revision, base image digest, and Python dependencies for reproducible builds.
  • Replace the global runner state with concurrency-safe, session-keyed state. Validate that /transcribe and /ws belong to the same Livepeer session, and prevent stale teardown callbacks from clearing a newer session.
  • Add bounded queues, strict audio and settings validation, meaningful backend health checks, and explicit error propagation so incomplete transcription cannot appear successful.
  • Add automated tests covering mock execution, 14/14 lossless delivery, concurrent-session isolation, TLS verification, settings updates, and continuing paid execution.
  • Add authenticated hosted-signer support with bearer headers and automatic token renewal.