[Rack] [Noisebridge-discuss] network down this afternoon, an interesting guide for people who want to help when the network goes down

[Danny O'Brien]

On Mon, Jun 4, 2012 at 9:14 PM, John Adams <jna at retina.net> wrote:
> Danny,
>
> I wrote a pile of code that does exactly this sort of thing.  If you need to
> reset hardware when the internet goes down, this might help.
>
> First, find/buy an el-cheapo baytech power controller (like the RPC-3.) It's
> a power controler that you can control over the network. I see see these
> things on eBay all the
> time: http://www.ebay.com/sch/i.html?_trksid=p5197.m570.l1313&_nkw=baytech+rpc-3&_sacat=0
>
> Then, download software out of my Github repo, configure the power
> controller and using my check_dsl script, you'll have a took for instant DSL
> modem restarting should you lose network connectivity. That's here:
> https://github.com/netik/rpc3control
>
> WRT to DHCP, Configure two DHCP servers with DHCP Load balancing. Then, if
> you lose one server, you'll still be able to service the
> pool: http://www.ipamworldwide.com/dhcp-failover-a-load-balancing/dhcp-load-balancing.html
>

Thanks!

We actually have a bunch of failover stuff (to go between
monkeybrains/sonic), but there's been a fair bit of recent work on it,
and I suspect what we're seeing is some intermittent hardware issue
that's interfering with that. I defer to the ladies and gentlemen of
rack to actually describe what we have right now, but
https://www.noisebridge.net/wiki/Network#Routers

DHCP load balancing sounds a good addition, but i wonder if we might
be better off with thinking about how to simplify the network so that
it's fixable for humans when it breaks in obvious ways.

d.



> -john
>
> On Mon, Jun 4, 2012 at 9:02 PM, Danny O'Brien <dannyobrien at gmail.com> wrote:
>>
>> Summary: my suspicion is that our DHCP server is still on the fritz,
>> but we got sidetracked. see
>> https://www.noisebridge.net/pipermail/rack/2012-May/001543.html for
>> rebooting DHCP server
>>
>> 1. Somebody says to me "oh the Internet is down"
>> 2. I say, in a friendly tone, "Do I look like I'm in charge?"
>> 3. Person says, also friendly, "No-one is in charge, but you look like
>> somebody who might know someone who can get the Internet back up" <--
>> 10 anarchist hackerspace points!
>> 4. I loudly say "Is the Internet down for everyone?"
>> 5. Everyone mutters yes.
>> 6. I loudly say "who wants to help fix it?"
>> 7. General roar of acclaim, followed by hesitancy
>> 8. Me and (Paula?) start loudly troubleshooting the Wall of Tubes,
>> surrounded by people standing around trying to work out how to be
>> useful.
>> 9. I fail to make everyone useful. Minus ten anarchist hackers points to
>> me.
>> 10. Me and Paula decide to try powercycling the DSL modem, followed by
>> resetting the DSL modem, because it only has 3 lights green and the
>> Internet light is unlit
>> 11. This turns out to be wrong thing to do. Much of what I do next is
>> based on getting us back out of this error.
>> 11.5 I bug SuperQ about various things. He has a cold and is working,
>> but is helpful.
>> 12. Meanwhile, Hal turns up and notes that he can't get onto the
>> network via an AP.
>> 13. We work out that this is because he isn't getting an IP. Wired
>> devices are also not getting IPs.
>> 14. Hal learns about netmasks and /24s
>> 15. I call up Sonic and they are awesome. They explain how to fix DSL
>> modem. I leave note on modem instructing future noisebridgers not to
>> reset DSL modem, and that 3 lights is fine.
>> 16. Once that is fixed, people get on Internet again. I suspect DHCP
>> server magically resolved itself.
>>
>> I will try and think of ways we can detect when stuff goes down. It's
>> very hard for people in the space to know where to start, especially
>> when all the docs about the network are a) only on the internets, and
>> b) a bit out of date (are they out of date?). It's also a bit unfair
>> on both SuperQ and Jof to depend on them to troubleshoot this stuff.
>> Anyways.
>>
>> d.
>> _______________________________________________
>> Noisebridge-discuss mailing list
>> Noisebridge-discuss at lists.noisebridge.net
>> https://www.noisebridge.net/mailman/listinfo/noisebridge-discuss
>
>