[tor] sysctl tweaks to fix "Error binding network socket: Address already in use"

[Andy Isaacson]

I happened to notice that we were seeing a lot of this error on exit01:

==> /var/log/tor/notices4.log <==
Jan 09 00:12:59.000 [warn] Error binding network socket: Address already in use

The errors were coming from, according to strace:

15610 1326094578.702454 bind(3520, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("173.254.216.66")}, 16) = -1 EADDRINUSE (Address already in use)

The error was bursty, coming in great batches at varying intervals,
ranging from 10 to 60 seconds.  When bursting, they were across all
interfaces.

running "while :; do cat /proc/net/tcp | tail -1; sleep 1; done" while
the failure was happening, the number of TCP connections would slowly
creep up towards 77,000; then the errors would happen and the number of
connections would drop to 72,000 or so; then repeat.

I tried a few of the obvious possibilities from torservers.net's
suggested sysctls.  Increasing the local port list with
"ip_local_port_range = 5000 65000" didn't help, and neither did changing
the rmem / wmem parameters, but setting tcp_tw_recycle = 1 did resolve
the problem by decreasing the number of TCP sessions; we're now hovering
in the 45k - 55k range.

Open questions:

1. What codepath is giving us EADDRINUSE for an unable-to-allocate
error?

2. What resource are we exhausting?  77,000 is a very odd maximum value.

3. What tunable am I missing that needs to be increased?

Here's the full list suggested by torservers:

# optimizations
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432 
net.ipv4.tcp_wmem = 4096 65536 33554432  
net.core.netdev_max_backlog = 2500
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_fin_timeout = 4 
vm.min_free_kbytes = 65536
net.ipv4.netfilter.ip_conntrack_max = 196608
net.netfilter.nf_conntrack_tcp_timeout_established = 7200
net.netfilter.nf_conntrack_checksum = 0
net.netfilter.nf_conntrack_max = 196608 
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 15
net.nf_conntrack_max = 196608
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.ip_local_port_range = 1025 65530
net.core.somaxconn = 20480
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_timestamps = 0

-andy