Orchestrator compromise - Full Disclosure & Report

1

In the spirit of full disclosure and transparency to the community and delegates, we are sharing all the
details as we know them. This post will be broken into a few parts to make it easier to read.

PART 1/3

On Oct-23-2022 03:46:59 PM +UTC, the Sundara.eth Orchestrator wallet (0x1a196b031ea1a74a53ecbe6148772648e02f9d51) sent 0.03 ETH out to an unknown wallet (0xfb8c3ba8a46014400487f2fb4d539a5ff7bc367d) on Layer 1.

While this seems like a tiny figure, Titan-Node correctly pointed out, in effect, the wallet on L1 was cleaned out, a fact that seemed almost surreal to us having moved to L2 with funds still intact there.

Since this wallet was never interacted with manually (after moving to Layer 2 only the Livepeer binary uses it) it was, after careful review, unfortunately deemed to be compromised.

The issue was brought to my attention on Twitter by https://twitter.com/annhandt09 which I ignored as a poor scam attempt. The Twitter user dropped me an email explaining that his wallet lost approximately $600 worth of ParagonDAO tokens after my wallet sent 0.03 ETH to an unknown wallet which then sent him ETH post which he lost his tokens. He could search for details thanks to ENS and Twitter where Sundara.eth is linked in my bio. As English is not their first language, the tone of the email did not come across as genuine made worse by the fact that the email had two videos as attachments.
I took this as another attempt to ship malware embedded in a video and ignored it.
The irony of attempting this on a Livepeer Video Orchestrator was not lost on us.

It was only after they started posting on random Github projects and the Livepeer Discord accusing me of being a scammer that we actually bothered to look into the issue.
The fact that the Sundara.eth wallet held over 1 ETH on Layer 2, with 0.25 ETH and 71 LPT in the Livepeer contract which was not yet stolen lulled us into further complacency, no excuses.

We could not get ourselves to believe that a hacker would

  1. Clean the wallet on Layer 1 (approximately 0.03 ETH) but…
  2. Leave $2500 worth of ETH + LPT on Layer 2 untouched on the same wallet (WTF!)

It simply did not add up and we spent way too much time trying to figure out if either of the two partners with access to the wallet had accidentally mistyped an address and mistakenly sent funds to 0xfb8c3ba8a46014400487f2fb4d539a5ff7bc367d.

Today we withdrew all funds from the Livepeer contract and are in the process of rebonding to a new Orchestrator wallet and are informing our delegates of the issue requesting them to re-stake with the new wallet address: 0x5CaaaB7626eDc7123cF8484EdBC66a875DD32CC9 which will resolve to Sundara.eth shortly.

1 Like
2

PART 2/3

What security did we have in place prior to the incident?

This decision was made after a thorough review of the autonomous detection and response anti-malware, IDS, packet filter logs + discussions with older Orchestrator’s + my partner who had physical access to the Vancouver node where the encrypted JSON wallet and its password reside in plain text on a Full Disk Encryption NVME SSD running Debian Linux.

We have had extensive security in place, for example, even I cannot SSH in to these machines without just 1 specific Apple mobile device and 2-factor authentication from specific static public IPv4 addresses and the required ed25519-sk keys.
The node in Asia has a Stratum 1 GPS synchronized NTP server in the LAN for nanosecond precision timekeeping and a hardware random number generator for entropy.
We practice defense in depth, and use a multi-layered security approach but all the money and systems in the world cannot defend against human error, or a $5 wrench. More on this later in this post.

Although Etherscan / Arbiscan were configured to shoot an email on every transaction, complacency had set in where I missed that instead of the usual 0 Wei sent on Layer 2, this transaction sent 0.03 ETH on Layer 1 and no alarm bells went off. We are working on alerting when more than 0 Wei is reported in the Arbiscan transaction email and hope to share it once it’s working so all the O’s could benefit.

1 Like
3

Part 3/3

So how did we get compromised?

  1. Truth is we are not 100% sure as there are only two machines with the JSON and password required to access the wallet and we are fairly certain physical access to them was not the reason.
  2. The only other place where the JSON and password were stored was in a commercial VPS we tried testing in NYC that was a shared VM with a dedicated GTX1070. It was only two weeks as we never managed to get the VM speaking with the Livepeer network (never received a Ping Request) and in spite of them helping us troubleshoot, a solution was not found. Unfortunately, they did not let me wipe and rewrite blocks on the disk before shutting down the VM. They just dropped my partner an email stating our VM was killed since they had other customers who wanted the GPU. This did not raise alarms since they were quite nice and allowed us two weeks free of charge to setup and test and were responsive to our issue of being unable to connect to Livepeer.

I am not sharing the name of the host / business here since:

  1. I have no reason to believe they were malicious.
  2. It is more likely their host was compromised before they wiped my VM.
  3. The two nodes we physically own and control could be compromised and we are simply unaware of it even now. We are in the process of wiping them and starting from scratch all over again.
  4. I had actually used MEW (nub of me I know) to convert the Ethereum wallet to the encrypted JSON livepeer needs and they could be malicious or have been compromised unbeknownst to them.

Why were our funds on Layer 2 untouched?
This one has our heads spinning too and our best guess is that the wallet address and JSON password access was just fed to a bot that is unaware of Layer 2/3 contracts. We believe we dodged a bullet here, others have not been so lucky.

What are we doing to avoid this in the future?

  1. We have created a new Orchestrator wallet in a fresh OpenBSD VM using Metamask.
  2. This wallet will now hold just a few LPT to make it a smaller target.
  3. I will delegate the majority of LPT to my own Orchestrator using a hardware wallet (Trezor Model T).
  4. We will not be using the JSON / password in a flat file configuration for every O in each location. We will shift to using a Redeemer node or the dummy address configuration other O’s have tested successfully.
  5. We are figuring out how to grok Etherscan / Arbiscan alert emails to single out one’s for human review that do not meet the usual daily reward call / normal Livepeer transactions threshold.
  6. We will never put the wallet JSON / password on a VPS or a physical machine we do not fully control, ever, again
  7. Our Orchestrator ENS name had the same wallet set as the Registrant. Controller and ETH address. This is a poor security practice and moving forward the Registrant (aka owner) will be set to a cold hardware wallet. If you wish to transfer your ENS to a new wallet, the order in which you do things is important. Change the Registrant first to the hardware wallet and then connect with your new hardware wallet address and change the Controller followed by the ETH Address record to your Orchestrator wallet address.
  8. Since Metamask will likely never allow exporting a wallet to encrypted JSON, I used the popular and Eth developer recommended ethereumjs-wallet (Geth works too) NPM package to convert the Metamask private key to the JSON that Livepeer expects.

This is as easy as:

npm install ethereumjs-wallet

Create a file named key-export-json.js with the following content:

const fs = require("fs")
const wallet = require("ethereumjs-wallet").default

const pk = new Buffer.from(process.argv[2], 'hex') // replace by correct private key
const account = wallet.fromPrivateKey(pk)
const password = process.argv[3] // will be required to unlock/sign after importing to a wallet like MyEtherWallet

account.toV3(password)
    .then(value => {
        const address = account.getAddress().toString('hex')
        const file = `UTC--${new Date().toISOString().replace(/[:]/g, '-')}--${address}.json`
        fs.writeFileSync(file, JSON.stringify(value))
    });

Run $ node key-export-json.js <your-private-key> <some-random-password> and the JSON will be created in the same directory you ran the command.

Sincerely,
Avi / Strykar / Sundara.eth / Tomy

1 Like
4

Thanks so much for the full transparency and details on this. I’m really sorry this happened to you :frowning:

1 Like
5

Sorry for what happened to you Strykar :frowning: hope all your delegators will follow you on your new address because you deserve it.

1 Like