[Noisebridge-discuss] [Rack] network down this afternoon, an interesting guide for people who want to help when the network goes down

[Ben Kochie]

> On Mon, Jun 4, 2012 at 9:28 PM, Danny O'Brien <danny at spesh.com> wrote:
>> We actually have a bunch of failover stuff (to go between
>> monkeybrains/sonic), but there's been a fair bit of recent work on it,
>> and I suspect what we're seeing is some intermittent hardware issue
>> that's interfering with that. I defer to the ladies and gentlemen of
>> rack to actually describe what we have right now, but
>> https://www.noisebridge.net/wiki/Network#Routers
>
> For various level of "a bunch".
>
> Right now, we've got the two "modems" that connect us up to our ISPs
> (Sonic.net and Monkeybrains), and just one "router" that is doing the
> NAT translations as well as internal network services like DHCP and
> DNS relaying (the Vyatta-on-Soekris box).
>
> It's certainly feasible to make another Soekris with some redundant
> configuration, sans NAT state replication.
> It's usually pretty easy to fail over the stateless things (routes,
> next-hops), than things that track each connection or user (NAT
> translations, DHCP leases, etc.)
> The practical limitation of this is that connections will all break if
> things failover.

We could easily separate some of the services off of the one NAT box.

I've thought about setting up a synced virtual router on stallion using 
failoverd and vyatta's NAT state sync.

It would also possibly make sense to put the local DHCP/DNS services on a 
separate instance from the NAT handling.  We can easily do this with some 
virtual machines on stallion.  Or we could move some of these services to 
minotaur.

>
>> DHCP load balancing sounds a good addition, but i wonder if we might
>> be better off with thinking about how to simplify the network so that
>> it's fixable for humans when it breaks in obvious ways.
>
> What is complicated about what is there today?
> I realize that figuring out the practical thing that's needed when
> looking at the network is not always the most obvious thing.
>
> Perhaps some large labels about purpose on the hardware on the
> wall-of-tubes would help?
>
> --j
> _______________________________________________
> Rack mailing list
> Rack at lists.noisebridge.net
> https://www.noisebridge.net/mailman/listinfo/rack
>