[Rack] [Noisebridge-discuss] network down this afternoon, an interesting guide for people who want to help when the network goes down

[John Adams]

Danny,

I wrote a pile of code that does exactly this sort of thing.  If you need
to reset hardware when the internet goes down, this might help.

First, find/buy an el-cheapo baytech power controller (like the RPC-3.)
It's a power controler that you can control over the network. I see see
these things on eBay all the time:
http://www.ebay.com/sch/i.html?_trksid=p5197.m570.l1313&_nkw=baytech+rpc-3&_sacat=0

Then, download software out of my Github repo, configure the power
controller and using my check_dsl script, you'll have a took for instant
DSL modem restarting should you lose network connectivity. That's here:
https://github.com/netik/rpc3control

WRT to DHCP, Configure two DHCP servers with DHCP Load balancing. Then, if
you lose one server, you'll still be able to service the pool:
http://www.ipamworldwide.com/dhcp-failover-a-load-balancing/dhcp-load-balancing.html

-john

On Mon, Jun 4, 2012 at 9:02 PM, Danny O'Brien <dannyobrien at gmail.com> wrote:

> Summary: my suspicion is that our DHCP server is still on the fritz,
> but we got sidetracked. see
> https://www.noisebridge.net/pipermail/rack/2012-May/001543.html for
> rebooting DHCP server
>
> 1. Somebody says to me "oh the Internet is down"
> 2. I say, in a friendly tone, "Do I look like I'm in charge?"
> 3. Person says, also friendly, "No-one is in charge, but you look like
> somebody who might know someone who can get the Internet back up" <--
> 10 anarchist hackerspace points!
> 4. I loudly say "Is the Internet down for everyone?"
> 5. Everyone mutters yes.
> 6. I loudly say "who wants to help fix it?"
> 7. General roar of acclaim, followed by hesitancy
> 8. Me and (Paula?) start loudly troubleshooting the Wall of Tubes,
> surrounded by people standing around trying to work out how to be
> useful.
> 9. I fail to make everyone useful. Minus ten anarchist hackers points to
> me.
> 10. Me and Paula decide to try powercycling the DSL modem, followed by
> resetting the DSL modem, because it only has 3 lights green and the
> Internet light is unlit
> 11. This turns out to be wrong thing to do. Much of what I do next is
> based on getting us back out of this error.
> 11.5 I bug SuperQ about various things. He has a cold and is working,
> but is helpful.
> 12. Meanwhile, Hal turns up and notes that he can't get onto the
> network via an AP.
> 13. We work out that this is because he isn't getting an IP. Wired
> devices are also not getting IPs.
> 14. Hal learns about netmasks and /24s
> 15. I call up Sonic and they are awesome. They explain how to fix DSL
> modem. I leave note on modem instructing future noisebridgers not to
> reset DSL modem, and that 3 lights is fine.
> 16. Once that is fixed, people get on Internet again. I suspect DHCP
> server magically resolved itself.
>
> I will try and think of ways we can detect when stuff goes down. It's
> very hard for people in the space to know where to start, especially
> when all the docs about the network are a) only on the internets, and
> b) a bit out of date (are they out of date?). It's also a bit unfair
> on both SuperQ and Jof to depend on them to troubleshoot this stuff.
> Anyways.
>
> d.
> _______________________________________________
> Noisebridge-discuss mailing list
> Noisebridge-discuss at lists.noisebridge.net
> https://www.noisebridge.net/mailman/listinfo/noisebridge-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.noisebridge.net/pipermail/rack/attachments/20120604/5ed64fdf/attachment-0003.html>