[Rack] noisebridge.net sloooow

[Ben Kochie]

Ok, here's what I've done so far.

I've added memory cgroup support to most of the init scripts that matter.

There's now a script for checking memory use by process tree:
get-cgroup-memory-use.sh

11216 KiB - /cgroup/bind9/memory.usage_in_bytes
680 KiB - /cgroup/clamsmtp/memory.usage_in_bytes
4252 KiB - /cgroup/posfix/memory.usage_in_bytes
69432 KiB - /cgroup/mysql/memory.usage_in_bytes
80396 KiB - /cgroup/mailman/memory.usage_in_bytes
175448 KiB - /cgroup/clamav-daemon/memory.usage_in_bytes
51168 KiB - /cgroup/apache2/memory.usage_in_bytes

The last one output by teh script is the system total.
1048708 KiB - /cgroup/memory.usage_in_bytes

I've manually set a memory limit for apache2 to 500MB

This in theory should catch apache2 going over memory limits.  We can 
improve the memory cgroup settings in the future.

The one difficult thing is this, this does not work well for upstart jobs 
since the scripting for upstart is crappy.

-ben

On Tue, 19 Feb 2013, Ben Kochie wrote:

> Fucking crap.  I've started testing using memory cgroups to limit apache 
> memory use to keep it from blowing up the machine.
>
> I'm also testing adjustments to the oom killer to make apache the more likely 
> target.
>
> I'll likely create some scripts to automatically deal with this.
>
> -ben
>
> On Tue, 19 Feb 2013, Andy Isaacson wrote:
>
>> On Tue, Feb 19, 2013 at 01:10:15PM -0800, Jonathan Lassoff wrote:
>>> Somethings awful slow with the wiki again.
>>> 
>>> Halp.
>> 
>> Page loads were taking 10 seconds each.
>> 
>> memcached got OOMkilled.  Restarted, loads back down to 300 ms again.
>> 
>> Thanks for noticing.  Can we get an external latency monitoring system
>> up and running again?  Both TCP and HTTPS GET latency would be useful..
>> 
>> -andy
>> _______________________________________________
>> Rack mailing list
>> Rack at lists.noisebridge.net
>> https://www.noisebridge.net/mailman/listinfo/rack
>> 
> _______________________________________________
> Rack mailing list
> Rack at lists.noisebridge.net
> https://www.noisebridge.net/mailman/listinfo/rack
>