Guide: The most secure way to run an Orchestrator as of June, 2022

The issue:

A common concern among Orchestrators is the requirement to store our private keys on the same server as our node(s). This presents a security risk, especially when running multiple nodes with geo-dns. One way to get around this is to use a redeemer node, but this method is fairly inefficient and creates a single point of failure.

There’s a better way:

Through trial and error, a few Orchestrators, namely @NightNode have found a better way for us to secure our keys without the need for a redeemer node. This method allows for the complete removal of any keystore files from actively transcoding nodes, pertaining to the Orchestrators account, while still allowing those nodes to redeem winning tickets and operate without issues.

The guide:

This guide will go over how to set up a secondary eth account to use for redeeming winning tickets, while still pointing to the primary Orchestrators address. This guide assumes you already know how to set up a node and have a general understanding of how livepeer node operating works.

OS: Windows & Linux (we’ll be using Linux for the example).
Difficulty: Mildly frustrating but totally worth it.

Step 1: Generate a new eth account and private key.

There are a few ways to do this but the way that works for me is to run livepeer manually. On linux it’s ./livepeer <-flags>. By default, these new credentials will be stored in ~/.lpData/<network>/keystore although you can specify a custom location with the -datadir flag.

Create a password for the account when prompted.

Note: If you already have a keystore file set up on your machine, you may need to move it out of the .lpData directory in order for livepeer to generate you a new one.

Step 2: Create a fresh Metamask account.

You can do this by using the profile feature on Brave/Chrome.

Import the newly created keystore file into Metamask (this may take a while and can sometimes seem like Metamask is frozen. Just wait.)

Don’t forget to add the Arbitrum network to the new Metamask account.

If you’re on a headless server, you may need something like WinSCP to access your files so you can download the keystore file and upload it to Metamask.

There may be better ways to do this, but this is just what I do, personally.

So now we have a fresh eth account that we can access via Metamask. You’ll want to fund this account with a bit of aeth so it can redeem winning tickets.

Step 3: Add the -ethOrchAddr flag.

The key to making this all work is the -ethOrchAddr flag. In your config file or launch command, you’ll want to specify your Orchestrator address. This is the address listed for your O on the livepeer explorer. For example, in my config file for livepeer, I have it set up like this:
ethOrchAddr 0x9d61ae5875e89036fbf6059f3116d01a22ace3c8

So let’s recap quickly…

We’ve created a secondary eth account which we’ve added to a fresh Metamask account and funded with some aeth. We’ve also added the -ethOrchAddr to our config/launch command which means, in theory, we’re ready to give this a whirl and see what happens.

Step 4: Start your node normally.

There are two things you want to look for to make sure this is working.

  1. Make sure your O is using the eth account you created in step 1. It may help to point to this keystore file with the ethKeystorePath flag if it doesn’t find it automatically.

  2. You should see the address you lsited in step 3 show up as being active.

If everything checks out you can now remove any traces of keystore files pertaining to your primary eth account and let the secondary eth account handle all ticket redemptions. If you’re running a multi-node setup, you can simply copy the keystore file from your secondary eth account to your other servers. There’s no need to create a new eth account for every server.

We’re not quite done yet…

So this is all great, but there’s a slight problem here, which we can solve quite easily.

Throughout this guide we’ve solely been talking about redeeming winning tickets, but what about daily reward calls?

Well, in order to call rewards you must have your primary eth account/keystore file located on a physical machine running an Orchestrator node. However, this Orchestrator node doesn’t need to be transcoding so we’re going to circumvent this issue by running an Orchestrator node on a secure, local machine.

Again, it doesn’t need to receive streams, it just needs to be able to call daily rewards, which means you don’t have to open any ports.

The serviceAddr of the secure node should look something like this (any port will do):
serviceAddr localhost:9999.

If you don’t have an extra machine lying around, you can pick up something like this (may have wifi issues on linux) and run a node on there. You’ve essentially created a hardware wallet of sorts by doing this, securing your keys on one locked down machine while allowing all other nodes to operate independently. Thanks to @Titan-Node for coming up with this method.

There’s one con to running a node this way:

As it currently stands, there’s an issue with the explorer where Orchestrators using this method won’t see their winning ticket redemptions on the explorer history tab. To clarify, this is just an issue with the transaction history on the explorer. Your earnings still update on the explorer itself and payouts still appear in the orchestrator-payouts channel on discord and show as being won by your active Orchestrator.

There is an open issue regarding the transaction history which you can look at here.

That’s it, you can now run your Orchestrator node(s) knowing your keys are much more secure! Just make sure you always have a bit of aeth on your secondary eth account for ticket redemption.

7 Likes

This is amazing!
Truly the most secure way of running an Orchestrator node.
Also thanks for the link to an inexpensive standalone machine to call daily reward on.
A++ to @Authority_Null :raised_hands:

4 Likes

In addition to setting up a secure machine to host your keys, I would recommend using this python script by Nondzu to ensure it calls reward every round.

By default it checks if you’ve called every hour and calls it if not.

3 Likes

I’m not sure if this issue is related but I didn’t win a ticket all month. Today, I moved one of my nodes back to using my main eth keystore. Not one hour later I won a ticket on a different node that was still using a dummy account. The timing makes it hard to believe that something wasn’t going on with this method. Don’t know where to go from here in terms of testing. Multipe orchestrators have used this method with great success so i’m a bit confused.

I can now personally confirm that this method works as intended.

Just a quick update:
I can’t edit the original post but the issue listed at the end of the guide has been fixed in the new iteration of the explorer. Transaction history shows properly :+1: