RFP — Explorer Maintenance

RFP Applications
24

RFP — Explorer Maintenance — Retrospective

Introduction

The original objective of this RFP was to “restore the Livepeer Explorer to a secure, maintainable, and high-performance state while laying the groundwork for new network-wide data and governance dashboards.” RaidGuild redefined this objective for itself as 3 pillars of actions: clean, strengthen, and monitor.

For cleaning, we intended to fix critical bugs, close stale bug issue tickets, merge or close old and pending PRs, upgrade core packages, update the package manager, and remove unused or duplicate code.

For strengthening, we intended to add speed optimizations, fetch retries, unit tests, stricter linting, husky checks, and a CI pipeline for all PRs.

For monitoring, we intended to add uptime checks on pages, health checks on APIs, external dependency monitors, and longer-retention logs.

Finally, in the midst of these improvements, we were also tasked with shipping the “voting-transparency feature,” which had existed for some time as disparate, incomplete, and stale PRs.

Commitments Delivered

Milestone 1

Delivered

Adjusted

The audit phase took longer than expected because it involved synthesizing multiple reports, coordinating meetings, gathering feedback, and getting approval on the roadmap. While the team was able to catch up later, this initially put the project about 1-2 weeks behind going into Milestone 2.

Deferred

Nothing from this milestone was meaningfully deferred. In hindsight, though, it would have been better to budget an extra week for the audit, especially given the amount of human coordination involved.

Milestone 2

Delivered

Adjusted

The performance deliverable shifted from strict benchmark targets to automated performance regression monitoring. While baseline tests were recorded and a Lighthouse CI test was added, the team did not end up using hard performance thresholds as the main measure of success. Instead, the goal became preventing major regressions on future PRs and maintaining a historical record of dashboard performance.

The Lighthouse CI automation has been useful, but it has also caused some friction. In practice, it sometimes flagged PRs as regressive without a clear cause, leaving authors to either spend time investigating inconsistent results or ignore the warning. Overall, the automation is probably worth keeping, but it should be tuned for more consistent and useful results.

Deferred

The full data layer migration to The Graph was not completed because the remaining RPC calls require subgraph changes before they can be replaced. The team removed RPC fetches where possible, but the remaining work was moved outside the scope of this RFP.

Milestone 3

Delivered

Adjusted

The monitoring scope was narrowed after implementation began. While the improved logging, health check endpoint, uptime checks, and alerts were useful additions, several of the goals listed in the original monitoring issue proved to be more than the Explorer needed in practice.

Deferred

Most of the monitoring ideas beyond health checks and uptime checks were either scratched before implementation or determined to be unnecessary after further review. The final monitoring setup is intentionally lighter than the initial version of the plan.

Milestone 4

Delivered

Adjusted

No major adjustments were needed for this milestone. By this point, the work was mostly focused on stabilization, documentation, and preparing the Explorer for handoff and long-term maintenance.

Deferred

Nothing significant was deferred from this milestone.

Delivered Beyond Commitments

Since the format for the retrospective wasn’t created until after this RFP began, we didn’t track features and bug fixes in a way that makes it easy to discern exactly what went beyond agreed deliverables. That being said, we ended the auditing phase with a list of features we agreed to tackle throughout the RFP, and the number of completed items appears to go beyond what the RFP originally called for. The likely incomplete list of merged features is below:

On top of this, we also serviced any critical bugs that arose in the middle of working on other RFP tasks. The biggest that comes to mind was a Next.js bug that was flagged in December. While the fix itself was a fairly minor task, it had cascading effects leading to updates on how different pages were handling different components (server-side vs client-side), as well as how how some packages were transpiled.

This is one example, but there are certainly others that are a bit harder to dig up.

Commitments Not Delivered

While the team made a good-faith effort to address all commitments, there were a few items that were consciously left incomplete, or were postponed, primarily due to information that was learned in the process of tackling them.

The most significant incomplete item was the data layer migration to The Graph. We succeeded in removing RPC fetches where possible (the vast majority), but it was discovered in the process that the subgraph did not provide enough data coverage to replace everything. To quote this issue ticket that is tracking the remainder of the migration:

After migrating core Explorer data fetching from RPCs to The Graph subgraph in PR #327, a small number of Explorer endpoints still rely on RPC calls due to missing or incomplete subgraph coverage identified during that work.

Another area where the outcome was less clear-cut was performance benchmarking. In the first place, the baseline tests were likely made too late (after the initial batch of core upgrades), which meant it didn’t reflect how much the dashboard improved from day 1. Secondly, the Lighthouse tool doesn’t appear to reflect what users actually experience, and the results of the test can sometimes swing wildly. On the positive side these automated Lighthouse tests have left a paper trail of performance throughout the RFP, and can be used to track improvements and regressions moving forward. The limitation is that the stats should be taken with a grain of salt. It’s recommended that these tests be audited in the future for higher precision. On performance altogether, though, we have received anecdotal feedback that the site is faster than it was before the RFP.

Finally, it’s worth noting that not every bug and stale issue in the repo was closed out by the end of the RFP. However, all remaining open bug tickets were either opened after the RFP began or were purposefully postponed. Similarly with stale issues, while the majority of these were closed, some were intentionally left open after discussion during the auditing process.

Impact

Overall, the work meaningfully advanced the goals of making the Explorer cleaner, stronger, and better monitored. For cleanliness, we wanted the outcome to be a repo that felt active and well-maintained. This meant regular upgrades, pruned old code, and stronger contributor engagement. One useful indicator of this is the package.json file, which now shows the latest package version of most core libraries. But as listed above, there were also a number of PRs that removed old and duplicate code. We also had the good fortune of having @Roaring30s contribute to a number of pretty significant refactors and features.

For strengthening, the details of what we did are listed above, but the impact is that this groundwork led to our team only pushing 1 bug to the dashboard (as far as I’m aware; and of course, bugs may be found in the future). The caveat is that we did push a caching issue when hot-fixing the Next.js bug that was flagged in December, but this was under rather intense circumstances to address that Next.js vulnerability quickly.

For dashboard performance, there is anecdotal evidence of improvement, though the available benchmark data is less definitive. While performance stats are being tracked on every PR now, we have to rely on Discord quotes to see how things have improved since the start of the RFP:

The explorer is in the best shape it has been in a long while. very snappy and responsive.

Explorer looks amazing

Shout out to the Explorer team. I see noticeable improvements from explorer

Explorer looks really good lately, great ux updates!

The explorer improved a lot under Raidguild’s contributions

The dashboard also appears to be more stable than it was before the RFP. Before the RFP, the homepage would sometimes show indefinitely loading data charts. Since working with the API team on this issue (originally tracked here), I have not personally seen this issue recur. Additionally, we’ve been able to see great health stats on both the dashboard…

Homepage uptime monitoring
Homepage uptime monitoring2074×526 47.5 KB

And the subgraph…

Subgraph health check
Subgraph health check2050×522 46.2 KB

Key Learnings

  • Detailed scoping is important; Rick at Livepeer made this very easy with the original RFP write-up
  • Auditing before implementation work begins is a great way to determine if a medium-sized endeavor is actually worth pursuing; it is also a great way to track progress (or regressions) that have been made throughout the course of a project
  • Features and tasks that involve heavy human coordination (like the audit) often needs more than expected timeline cushion
  • Fast communication with Livepeer stakeholders made things go very smoothly
  • Throughout the 6-month period of the RFP, an enormous shift in developer tools and strategies took place; it’s very likely that future work could be handled at a higher velocity
  • More direct engagement with delegators would have improved intuition about what to prioritize
    • It’s recommended that more time be spent in the Livepeer Discord engaging with users directly

Conclusion & What’s Next

While RaidGuild’s official commitments are concluded, it plans to remain active in the community, both in the Explorer repo, and in the occasional Water Cooler meeting. Livepeer has been a pleasure to work with throughout this project (from the beginning with a very clearly defined RFP), and RaidGuild hopes to continue working together in the future.

What RaidGuild is primarily handing off is a project roadmap for the next few months, as well as a series of milestones that make up the larger project. As RaidGuild is not officially committed to future work as of now, these handoffs are merely recommendations. If official work does continue, however, this roadmap will certainly be wrapped into it (or at least, it will be proposed to be wrapped into it).

This project benefited from strong collaboration between @Beyondr, @jip, and @sasquatch at RaidGuild, the guidance of @Rick and @Mehrdad at Livepeer, and the outside contributions of @Roaring30s.

2 Likes