[Noisebridge-discuss] network down this afternoon, an interesting guide for people who want to help when the network goes down

[Jonathan Lassoff]

On Mon, Jun 4, 2012 at 9:28 PM, Danny O'Brien <danny at spesh.com> wrote:
> We actually have a bunch of failover stuff (to go between
> monkeybrains/sonic), but there's been a fair bit of recent work on it,
> and I suspect what we're seeing is some intermittent hardware issue
> that's interfering with that. I defer to the ladies and gentlemen of
> rack to actually describe what we have right now, but
> https://www.noisebridge.net/wiki/Network#Routers

For various level of "a bunch".

Right now, we've got the two "modems" that connect us up to our ISPs
(Sonic.net and Monkeybrains), and just one "router" that is doing the
NAT translations as well as internal network services like DHCP and
DNS relaying (the Vyatta-on-Soekris box).

It's certainly feasible to make another Soekris with some redundant
configuration, sans NAT state replication.
It's usually pretty easy to fail over the stateless things (routes,
next-hops), than things that track each connection or user (NAT
translations, DHCP leases, etc.)
The practical limitation of this is that connections will all break if
things failover.

> DHCP load balancing sounds a good addition, but i wonder if we might
> be better off with thinking about how to simplify the network so that
> it's fixable for humans when it breaks in obvious ways.

What is complicated about what is there today?
I realize that figuring out the practical thing that's needed when
looking at the network is not always the most obvious thing.

Perhaps some large labels about purpose on the hardware on the
wall-of-tubes would help?

--j