[Noisebridge-jobs] seeking work: Sr. Unix System Admin / Operations Engineer

Tue Apr 27 18:07:57 UTC 2010

Hi,

I'm interested in a Systems Administrator position that provides me
with daily challenges
building highly available infrastructure and scaling systems.
I am currently available to work in San Francisco, Berkeley, Oakland
or Emeryville.
I grew up in San Francisco and I have over 11 years experience as a
Unix system administrator/Systems Engineer
for ISPs, Software companies and various mission critical websites...

My resume below can also be found here:
http://www.lumiere.net/~mrdavid/resume/
While working for Spinn3r.com (a small company with high performance
requirements!)
I was responsible for the companies entire infrastructure and 24/7 uptime.
Some of the daily challenges involved scaling database systems
and creating highly available systems such as loadbalancers, DNS
servers, API servers etc...
I also established many best practices and policies.
This included an on-call rotation, monitoring systems such as Nagios
and Ganglia,
automating server and software deployment using BCFG2 (cluster
management software like Puppet)
with Debian packages... I prefer to automate everything!

More recently I've been using an in-house cluster configuration
management solution at Raptr. I am definitely a proponent of
automation and cluster configuration management systems!

While at Spinn3r I worked on many critical operations projects including a live
datacenter migration as well as various software projects like patching Mysql;
I isolated two of Google's Mysql V2 features
and submitted the patches to the Ourdelta mysql project (these two
links should have my name) :

http://ourdelta.org/docs/kill-if-idle
http://ourdelta.org/docs/innodb-freeze

Also my boss at Spinn3r, Kevin Burton encouraged me to blog about my
sys admin work which I did here:
http://david415.wordpress.com/

At Spinn3r I learned Python on the job (I'd previously automate stuff
using Perl); these were the first programs I wrote to automate Mysql
database operations :

http://bitbucket.org/david415/mysql-cluster-tools/
http://bitbucket.org/david415/ganglia_gmetric_scripts/

I also posted to the Ganglia developer mailing list commenting on the
design flaws of the gmond python module interface :
http://www.mail-archive.com/ganglia-developers@lists.sourceforge.net/msg04971.html

In my free time I am often involved with Noisebridge.
We are a non profit 501c3; an infrastructure provider for technical
and creative projects in San Francisco :
https://www.noisebridge.net/wiki/Noisebridge

Although most of my database experience is with Mysql, I've recently
taken on a project involving PostgreSQL; I'm setting up Postbooks for
my friend's business, B. Spoke Tailor:
http://www.bspoketailor.com/

Cheers,

-- David Stainton

DAVID A. STAINTON's Résumé
dstainton415 at gmail.com
415-205-6020

Work History

    Operations Engineer Raptr, Mountain View (January 2010 - April 2010)

        Some projects I worked on:

            * rolling upgrades to the Mysql clusters
            * operations and capacity planning of Mysql clusters...
            * database operations automated via Python
            * setup OpenVPN so my co-workers could more easily gain
access to the office and datacenter
            * deployed a cluster of Ejabberd nodes for scalable IM solution
            * assembled various Sun servers with superior RAID
controller cards and additional drives, CPUs and memory

    Operations Engineer Spinn3r, San Francisco (2008 - August 2009)

        I was responsible for:

            * 40+ servers with hardware managed-hosting (i.e. not
colocation and not managed-hosting)
            * rolling upgrades to the Mysql clusters
            * live migration from ServerBeach to Softlayer managed hosting
            * operations and capacity planning of a large Mysql clusters...
            * database operations automated via Python
            * continually improving monitoring systems (Nagios and Ganglia)
            * on-call rotation
            * cluster configuration management with BCFG2 (similar to Puppet)
            * automatic Debian package creation

    Linux Systems Administrator Snapjot, San Francisco (April 2006 -
August 2007)

            * Network Administration: fixed network outages, setup
loadbalancing traffic to webservers via HAProxy and STunnel,
customized IP security policies and routing schemes with a Juniper
Netscreen firewall, Linux IP Tables and Cisco routers
            * Systems: "shadow DNS" via Daniel J. Bernstein's TinyDNS,
mail via Postfix, SpamAssassin and qpsmtpd with a custom Perl plugin,
Apache, Tomcat, MySQL, Squid, Mon (service monitoring system) w/
custom Perl scripts, backup system via cron scripts
            * Applications: VMWare Server setup with Windows and Linux
images, Cruise Control - Java continuous build server

    Site Operations Engineer Snapfish, San Francisco (November 2004 -
August 2005)

            * Worked on call, responsible for site reliability and uptime.
            * Facilitated in site downtime maintenance including a
timed bounces of various services for site operation, collaborating
with the DBA etc...
            * Wrote various monitoring and database analysis tools in
Java, Perl and Bash. This includes usage of Gnuplot and Perl to
generate web viewable graphs, Mon monitor's which send alerts to the
pager, a Java program to automatically fix failed website orders...
            * Ran many SQL queries on various Oracle database nodes to
maintain coherency for Snapfish accounts and photo albums.
            * Performed various systems administration tasks such as
maintain crontabs, configure postfix, custom compiled apache etc.
            * Responsible for the management and generation of all
internal SSL certs. Used perl to automate the process.
            * Reported upon site bugs and collaborated with the site
developers to get them fixed.

    contract Tech Support Engineer / Developer, Barracuda Networks,
Cupertino (May 2004 - July)

            * Serviced tech support calls troubleshooting firewalls,
routing problems, DNS and mail server configurations.
            * Automated tasks in perl and setup SNMP monitoring.

    contract Software Quality Assurance Engineer, DataDomain, Palo
Alto (January 2004 - April)

            * Performed hardware/software manual tests.
            * Wrote Perl programs to automate testing of DataDomain's software.
            * Installed and configured Linux systems to run QA tests
of DataDomain's backup server.
            * Debugged program crashes/core dump analysis with GDB
(the GNU Debugger).
            * Created performance tests to ensure the Data Domain
Restorer met product SLA.

    Network Tools Developer, Northpoint Communications, San Francisco
(April 1999 - December 2000)

            * Deployed custom monitoring software to Solaris/Linux
servers in less than 50 points of presense...
            * Wrote tools to automate the monitoring and maintenance
of the network. Some of these tools were used to modify the
configuration of hundreds of Cisco routers, Ascend switches, automate
configuration of MRTG (multi router traffic grapher) etc.
            * Worked in a project writing in-house software to
automate the testing of copper loops (for DSL) from circuit orders. I
programmed the backend code and collaborated with database and web CGI
programmers.
            * Configured CopperMountain DSLAMs, Cisco routers and
Ascend frame-relay switches

    Network Engineer / Tools Developer, CRL Network Services, San
Francisco (December 1997 - October 1998)

            * worked with hundreds of cisco routers
            * Wrote various monitoring facilities for the Network
Operations Center, including a program which displayed packet loss to
peer networks from core routers
            * Worked extensively with MRTG (multi router traffic grapher)
            * Performed frame-relay trouble shooting on Ascend
switches (via HPOpenView) and EIGRP/BGP trouble shooting on Cisco
routers.
            * Set up Cisco Flow Analyzer, MIT Kerberos V4 on a SunOS
server, including serious Sendmail configuration, modified the C
source code for the standard local mailer tranfer agent to include
Kerberos authentication and modified the C source code of the Merit
AAA RADIUS server for special username parsing.

    Skills

        Operating System Environments:

            OpenBSD, NetBSD, FreeBSD, Linux (Gentoo, Ubuntu, Debian,
Redhat, Fedora Core, Centos etc.), Solaris, SunOS, BSDi

                    I have extensive experience configuring/compiling
the Linux kernels and have written my own kernel patches and kernel
modules. I have some knowledge of Linux kernel internals. When I used
Solaris often I'd create customized Solaris packages for cluster
software deployment.

        Programming Languages:

            Python, Perl, C, C++, Java and Bash

        Misc. Software:

            Puppet, BCFG2, Apache, Tomcat, Qpsmtpd, Postfix, Daniel J.
Bernstein's TinyDNS, BIND, PowerMTA, Squid

            RedHat Kickstart, Cobbler provisioning system
            Loadbalancers: BigIP, loadbalancer.org, pen, balance,
HAProxy, Foundry
            Authentication services: Merit AAA RADIUS server, MIT
Kerberos, NIS, OpenSSH, openLDAP
            Monitoring Software: Mon, Nagios, Ganglia, Cacti, MRTG

        Education:

                * Summer 2001 University of Virginia,
                  Tibetan language

                * 2001 - 2002 Sera Jey Monastery, south India,
                  Tibetan language and culture

        references available