[tor] pushing 336 Mbps

[Andy Isaacson]

On Fri, Oct 14, 2011 at 08:05:23PM -0700, Michael C. Toren wrote:
> >  - we're running on Xeon X3350  @ 2.66GHz, quad core.
> >  - according to oprofile, we're spending around 25% of total CPU time in
> >    AES.  If we had software with AES-NI support, and a AES-NI CPU, we
> >    could cut that down to 5% or so.
> 
> IIRC, tor does not have AES-NI support?

It seems to be calling into OpenSSL for AES.  OpenSSL doesn't have
AES-NI in a stable release last I checked -- it's in the 1.1 branch.

> If it did, though, is the correct takeway from the above that if we had
> an AES-NI CPU, and if tor support AES-NI, we could push 20% more traffic
> (~90 Mbps) at peak?  

It's not entirely clear.

The CPU peaks are definitely *NOT* tightly correlated with the bandwidth
peaks.  BW is extremely periodic, CPU is much more variable.  Sometimes
we'll have the CPU maxed out doing only 330 Mbps, other times we are
pushing 400 Mbps at only 90% CPU.

As a specific example, for a 8 hour period on Thursday night we were
pushing 350 Mbps +- 10 Mbps.

Over that same time period, CPU utilization varied from 280% to 340%,
with a fairly smooth slope visible on the CPU% graph.

So presumably there's some overhead (connection setup, TCP stack
bookkeeping) or external variable (number of circuits created per
minute, amount of TCP retransmit going on) that is affecting the CPU
usage, above and beyond the known expensive parts of running Tor.

-andy