Adventures In Geekdom
I am a sysadmin and programmer (in fact, this website is a wiki I wrote in Ruby/Rails) looking for a challenging, rewarding position, ideally in large-scale systems administration. This page records some of my more significant exploits and learning experiences in those areas, and gives a good idea of my skills and attitude towards technology and learning. You may also be interested in my traditional Resume. Please feel free to Contact Me about any job openings you might have, or about anything else on this page.
As a general note, a lot of these stories took place while I was working at Pulsar Aviation, a tiny company (around 3 full-time people) with very limited resources that does maintenance support for commercial aircraft, and has offices on the decommissioned Norton Air Force Base on the edge of, well, not nowhere, but not much either (specifically, in San Bernardino, California; an hour or so east of Los Angeles County).
Figuring Out Verizon's DSL Protocol
Smart geeks know that we should read the documentation, but sometimes even having done so won't keep you out of trouble. There are times when, even after you've carefully prepared by reading as much documentation as you can find, it doesn't help, and you find yourself reading raw ATM cells in hex, trying to figure out why your modem/router can establish a link, but won't talk IP over it.
Learning BIND9 In A Hurry
I love working with technology and learning new things, but there's "Hey, I'll play around with X until I've learned it" and there's "My choices are learn X right now, or tell everyone tomorrow morning why they can't get to the mail server". The latter is a lot less fun, but it sure is motivating. In this case, X was BIND9, and it had been a 12 hour day by the time I realized I was going to have to make it work, and I'd never edited a raw zone file before.
"Helping" My Dad Install His HD
So, when I was 10 years old (1994), my dad got a shiny new hard drive for his 486DX computer. He, as a geeky sort of guy, was really excited about having so much more space and wanted to install it as soon as he got home. But when he brought it home and tried to install it, his computer refused to interact with the new drive. He swore up and down and rebooted over and over again trying different options to get it to work.
Finally, he got pissed off enough that he walked out of the room to calm down. I, as an even geekier version of my father, walk over to his computer and start trying to figure it out. When he comes back a few minutes later, I calmly inform him that "I fixed it for you, Dad." He blinks, then looks at the screen where I point to the window displaying his new HD.
"How'd you fix it?"
"Well," I say, moving back to the open tower case next to the desk and pulling the IDE ribbon a little higher up. "You had this cable plugged in backwards. The red line needs to be on this side."
Learning POP3
When I was 16ish, I was using BeOS R5 as my daily operating system. One day, I began having trouble with my e-mail client (BeMail). It'd get about 20 messages into a download, and then crash, leaving all those messages still on there, and unable to get past the one making it crash. I looked up the RFC and learned enough of the protocol to use telnet to log into my POP3 server and delete the message causing the issue.
Fun With Beta Kernel Releases
Around 2003 (I was 19 or so) I'd long since gotten computers of my own and begun to mess around with various alternative operating systems. At this point, I'm running Debian Woody as my day-to-day OS and I'd finally gotten around to setting up an older HD as a backup drive, so I felt more safe playing around with the betas of Linux 2.6 that had recently been released. I grabbed the tarball for 2.6.0test2 and compiled it. Next I tried to compile the nvidia module, expecting it to fail completely, which it did. After hunting down and applying a user-supplied patch, I got things back on track and went for my first reboot into 2.6, where I saw "Booting kernel..." followed by... nothing. Crap.
In the kern.log, I discover that the system was actually active well past the point where it had printed the "Booting kernel..." message. It found and activated my hardware, and later on I can even see evidence that init had started and mounted the filesystems and launched at least some of my daemons.
I compare the log with the successful boot on 2.4 and fail to find any important differences. So I figure that I'll see if it's a problem with the framebuffer driver.
Looking for corroboration for this theory (or for something to suggest another problem) before I start digging into that, I grep all of the logs starting from the point where I saw the messages from init. Here I found that X repeatedly tries to start, but fails, /dev/tty[1-6] can't be opened and module 'serial' can't be found. A-ha!
Looking through make menuconfig, I hunt down the serial module and also discover a bunch of new "drivers" that had apparently been separated out from the core kernel during the shift to 2.6. These modules are also required for the use of consoles. After playing around some more, I also find out that it's the inclusion of the framebuffer driver that's preventing X from starting. Now I can get the system up to the point where I have a GDM login screen, but it's not responding to the mouse or keyboard. Going back into make menuconfig, I find that these functions have also been separated out from the core kernel. I enable them, recompile/reboot and then bask in the glory of my shiny new kernel.
The Case Of The Apparently Firewalled Connections
Sometimes it's difficult to tell when a connection issue is caused by our ISP or when it's something within our network. (Or something else entirely, like the time a work crew decided that they were going to rewire all of the phone lines going to their part of the building and took a pair of shears to the bundle that connected much, much more than just their portion.) On top of that, even when you know where the problem is, that doesn't necessarily make it any easier to solve, which is what led me to poring over tcpdump output in which both sides of the connection were sure they were sending all the right things, but weren't seeing what the other side said it was sending ... but only on IPv4, IPv6 worked fine.
Playing With IPv6
As the earlier story about trying the Linux 2.6 beta releases demonstrates, I'm the sort of geek who will bust open and play with the latest toys just for the sake of trying it, especially if I think it's the sort of thing I'll need to know about in the future. To that end, I'd played around with IPv6 tunnels here and there and given a few spare moments last year, I began the process of putting our office on the IPv6-ternet, starting with just a tunnel to our web/e-mail server, and eventually getting a /48 and putting up a radvd.
To date the only systems on our network that are not on IPv6 are the networked scanner and our iPhones. (I've no idea why Apple has decided against supporting IPv6 on the iPhone thus far. They have pretty good support for it everywhere else.) All of our server software (well, the ones for which IPv6 is relevant) is configured to respond to IPv6 requests. I also have our DNS server set up to route requests for google.com and its subdomains through the DNS server of our IPv6 provider, who is set up with the Google over IPv6 program.
A few things I learned during this project:
- Most modern Operating Systems support IPv6. Mac and Linux support it out of the box, as do Windows Vista and 7. Windows XP (SP2+) supports by adding it as a protocol to a network interface.
- DHCPv6 exists, although support for it is touchy. Mac OS X doesn't support it at all, relying solely on either static configuration or the standard IPv6 autoconfiguration via the Neighbor Discovery Protocol. Specification of a DNS server is possible in a router advertisement (NDP), but it's a recent addition to the protocol and none of the clients seem to understand it. (Meaning that I can't quite turn off IPv4 and have things work automagically.)
- Windows, by default, ignores the usual autoconfigured address mechanism and generates a random host ID periodically for "privacy". Disabling this is two lines on the command prompt.
- Reverse DNS isn't much different than IPv4 rDNS, although it's more granular, offering the ability to delegate after each 4 bits.
Complex NAT Over IRC
Like many geeks, I have a tendency to immediately start trying to solve any problem someone happens to mention in my presence.
In mid-May 2009, a friend of mine (who is, for the record, a Senior Sysadmin, and at the time had been one for more than half a decade) was on IRC, talking about trying to properly set up his home router (which had a Linux-based firmware mod on it) to handle his complex (for a home user) addressing scheme. He has four static addresses, three of which are for specific systems; the last is used for everything else.
He'd copied a configuration for "One-to-One NAT", and had some success with it, but experienced breakage attempting to modify it. After I started trying to fix that, he said "If you're going to spend a bunch of time looking at this, there's a much more important problem."
The much more important problem was the same problem I'd had with my Verizon-branded router the year before: Packets from inside the NAT addressed to a system behind the same NAT via its external address were being dropped.
I had him dump his iptables configuration to a pastebin and looked it over, discovering that the rule handling NAT destination address munging was restricted to only packets coming in from the outside address. I had him adjust the configuration to remove that restriction, after which the problem was fixed for all the static NAT entries, but not for communication between those machines and the dynamic NAT machines.
A little more playing around, and we discovered that it was because there wasn't any Source-NAT in place for packets originating from and destined for the local network. I gave him the correct iptables line for that, and the happy dances commenced.
Incidentally, the original problem that got me involved fixed itself during all this; he credits me for that, which is fine by me.
Other Things I've Learned
- Linux 802.11 devs love to change the location of firmware between versions, much to my annoyance.
- Sometimes, the wrong driver grabs your device at boot, requiring manual intervention. (Again, the 802.11 devs must have it out for me, as this seems to happen much more often with their drivers.)
- A file is not the same thing as an inode; remembering this will sometimes be a life-saver.
- On that note, deleting a file does not necessarily delete the inode behind it. lsof is awesome.
- Also awesome: strace (and ltrace!), netcat, tcpdump, find, xargs, fail2ban (beyond its stock confguration; you can do really neat things with it)
- Knowing that you can pass "init=/bin/sh" to the kernel boot line can also be a life-saver.