Increase File Limit as a Transcoder

#1

If you are running a transcoder on a linux box, sometimes you’ll see a “too many files open” error. When this happens, the transcoder can stop doing work.

This is because the Livepeer transcoder runs an internal IPFS node. Since IPFS requires a larger file limit, you have to change your default file limit. For Ubuntu users, you can do this by:

  1. Change /etc/security/limits.conf by adding * hard nofile 4096 and * soft nofile 40960
  2. After that, change /etc/pam.d/common-session by adding session required pam_limits.so
  3. Restart the machine, and verify via ulimit -a to make sure the soft/hard limits have changed.
#2

Should that be 4096, rather than 40960?

#3

4096 is usually not big enough. Some people suggest 10000 to 20000. I used 40960 to be safe.

#4

Thanks, I wasn’t sure if that was a typo :slightly_smiling_face:

Transcoder Campaign: Chainflow Transcoder
#5

I was trying to get around the db write errors by changing the value on osx

seems like it isnt something you can do anymore there?

ulimit -u 2048

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1418
virtual memory (kbytes, -v) unlimited

#6

wouldnt it make more sense to limit the ipfs connections instead of just allowing more?

in the end by running the ipfs node we are adding a unpredictable load/cost to the transcoders that could be minimized by using a private IPFS implementation as yahya mentioned,

or completely removed/made optional by using the rpc connection instead as josh mentioned.