Looking back 30 years as a sysadmin

The profession — and everything it involves — has changed dramatically, but has been (and still is) a fun ride.

Looking back after spending more than 30 years as a Unix systems administrator, I have to say that’s it’s been quite a ride.

It certainly wasn’t 30+ years of doing the same thing. Instead, the technology and the job have gone through incredible changes along the way. There were dramatic improvements in the hardware that I managed and always plenty of new tools to learn and use.

Over the years, I went from reveling in how much work I could get done on the command line to grappling with some big issues — troubleshooting some very complicated problems and figuring out how to best protect my employers’ information assets. Along the way, I worked with some amazing individuals, got laid off (once), and learned a lot about what works and doesn’t work both from a technical and a career perspective.

Here are my reflections on the changes I’ve seen and those still to come.

How the technology has changed

In the earliest part of my career, I actually used keypunch machines — first, when processing payrolls for client companies while working for a large New York City bank (and putting myself through college) and second, when taking my first programming class. At the bank, I built punch card “programs” to make it easier for the keypunch operators to jump to the next field for the data they were entering. At the college, the class was an introductory programming class based on Fortran. Yes, Fortran. The following semester, the keypunch machines were no more and big clunky terminals took their place.

Keypunch operators
In college I had learned languages like Fortran, LISP, ALGOL, and Pascal. And, in one class, I built a simple operating system on a PDP system using assembly language. I remember “reading” the lights on the front of the system and how exciting it was when the attached printer spit out a sheet of paper as instructed. I’ve used many other languages since — like C and some Java, but I’ve mostly worked in scripting languages like sh, csh, bash, ksh, Python, and Perl. One of the most surprising things is how many languages have been introduced since I started in the field. The number of languages available seems to have increased maybe 20-30 times. This list from 2013 is probably no longer up-to-date: 256 Programming Languages

I remember in the early ’80s having to know the topology of hard drives in order to add them to my systems. Today, the systems are able to identify peripherals with very little work on my part. The number of cylinders, heads, and sectors … I had to describe the disk in these units for the system I was working on to be able to use the drive.
ALSO ON NETWORK WORLD: 8 ways to jumpstart your career

From the early ’90s, I still have somewhere a 300 MB (yes, that’s megabytes) disk that’s roughly the size of a shoe box and sometimes stare at my USB (“thumb”) drives, knowing that some hold as much as a terabyte. What an incredible comparison! If this trend continues, we’ll soon find that dropping a storage device on the floor will mean we’re no longer going to be able to find it.

I also remember backing up my servers using a reel-to-reel tape drive. The tapes were huge and they didn’t hold all that much data. Some of my file systems required 3-4 of them. Today, we use robotic tape drives and tools that automate the backups and keep track of what files went to what tapes so that you can restore files from various backup tapes with ease. And some of the backup technology today uses clever “deduplication” technology to reduce the size of data dumps by avoiding storing duplicate data, often reducing the size of backups to a small fraction of their original size.

robotic tape library
Of course, almost nothing has changed the field in which I’ve spent the last 30+ years as much as the Internet and the web have. When I worked for the federal courts, the district courts were connecting to systems in Washington, DC using a service called “Tymnet” which used packet switching technology. My project would not only put “minicomputers” (systems about the size of a college dorm refrigerator) into the courthouses, but make it so that all activity no longer had to push bits to Washington DC and back.

The growth of the Internet made connecting to arbitrary systems around the globe not only possible but common. And the introduction of the web (nee the “world wide web”) meant that I could find answers to many of my technical questions without even having to pull a book off the shelf. Today I often find myself wondering how I ever found answers to my technical questions before Google and similar web searching tools made their appearance.

How the jobs have changed
In general, the networks we manage are larger and more diverse. We won’t see the AppleTalk network segments that I remember from the ’80s doing their own thing, but the systems we run on our desktops and support in our data centers can be surprisingly diverse. More of the work we do is centrally managed through network services like NFS, NIS, DNS, etc.

Virtualization has become a major factor in our data centers. Many of our servers are now just segments of resources on larger servers, able to be shrunk and grown as needed to meet our demands, and easily migrated to alternate data centers as needed. On top of that, what now seems the ultimate virtualization — moving systems and even complete data centers into the cloud — dramatically changes what we as sysadmins are able to control and what we are responsible for.

Most of us rely on fairly versatile ticketing systems to keep track of all the problems that we are addressing and tasks waiting to be completed. We might be “just” doing systems administration, but that role has moved heavily into managing security, controlling access to a wide range of resources, analyzing network traffic, scrutinizing log files, and fixing the chinks on our cyber armor.
Security

In the early part of my career (maybe the first ten years or so), security was fairly lax. Maybe we forced our users to change their passwords every year. I remember once writing a program to pseudorandomly generate passwords by clumping two short words together, but it was nothing like what I do today. Security in those days was not a hot item and most of the people that I worked with were far more cavalier than I. When one of our speakers at a Sun User Group conference that I helped organize in the 1990s suggested that we all think like attackers, the thought seemed quite revolutionary.

This aspect of being a sysadmin has undergone more change than likely any other. Today, you’re irresponsible if you’re not behaving in a manner that might have seemed paranoid 20 years ago. The tools we use and the measures we go to in order to secure our systems are orders of magnitude beyond anything we would have considered back then. Passwords are longer and the systems we manage allow us to configure complexity measures. The suggested password length has gone from 7-8 characters to 12-14 and the expiration times have gone from once a year to once every three or four months for most of us.

In addition, the tools that we use have become dramatically more sophisticated. To some extent, we do become the attackers, using vulnerability testers like Nessus and Nexpose that discover the holes in our systems (hopefully before our attackers do) and sometimes even exploit them. We’re also on the lookout with intrusion detection systems watching for signs of malicious activity and data loss prevention tools trying to keep our organizational “jewels” from leaking out the back door. And following a briefing with Palo Alto just yesterday, I don’t imagine that I’ll ever think of firewalls in the same old way again. They’re moving from the perimeter of our organizations into the middle of everything we do. They’re smarter, faster, and they’re focused on what’s happening, not just on what doors (i.e., ports) the traffic is moving through.

How our communities have changed
In 1980, the Apple II computer that sat on a desk in the corner of my dining room had my neighbors thinking that I was a complete freak. To hear them talk, you’d think I had a centrifuge on my kitchen counter. And it was not because the computer was in the dining room or because it was an Apple. It was a computer and why I would have one sitting in my house had them looking at me really funny. Yet it wasn’t that many years before anyone without a home computer was considered weird. And these days, we’re all pretty much using wireless networks and probably everyone in the household has their own computer. Big change — even without mentioning all the other electronics that are practically mandated by our modern life styles.
Career choices

The downside: Compared to many IT jobs, there’s not much climbing up the corporate ladder for sysadmins. As a systems administrator, you’ll seldom be in the spotlight. You can easily still be a “bottom rung” (nobody reporting to you) worker after 30 years in the field. It’s also hard sometimes to get a sense of value. You generally get noticed least when everything is running smoothly. Unless you resolve Big Problems, most of the people you support won’t think about you very often. Maybe not even on Sysadmin Day.

Systems administrators are rarely customer-facing unless you count as customers the staff that use the systems that you keep humming along. And, even then, the big changes that you make are likely done after hours when everyone else is off duty and having a relaxing weekend or enjoying happy hour at the local pub. Do your job really well and no one will remember you’re there.

The upside: The work is seldom boring and there’s always something new to learn — something breaking, some new coming through the door. Even after 30+ years, the work is anything but monotonous. And the job pays reasonably well. There’s also a lot of variability in what you do and what you specialize in. You might automate all of your tasks or manage a huge data center, but there will always be something that challenges you and problems that need your attention.

Some of the significant trade-offs involve the kind of organization you work for. I worked in one company with only three employees and two independent contractors and other organizations with staffs of tens of thousands. The benefit of the smaller staff positions was getting to touch nearly everything and being involved in almost every aspect of the work. The big ones offered more chance of moving around and changing my organizational role fairly dramatically.

How much variability there is in your work depends on many factors, but I generally prefer having enough flexibility that I’m always doing something that I do easily and well and something else that is new and exciting. The mix keeps me feeling that I’m earning my keep and equipping myself for future challenges and opportunities.

The best jobs
For me, the best jobs that I’ve had involved my feeling that what I did was important. My stint working in the federal government was one of those because I knew that the analysts that I was supporting were helping to ensure that good decisions were being made on the national level. It was rewarding just to be a part of that.

At another (Johns Hopkins University), I managed the systems and the network for one department (Physics and Astronomy). The big plus was that I worked with some of the most brilliant people I’d ever have hoped to know — some trying to map the cosmos and other peering into the nature of the tiniest subatomic particles — and the students who helped out from time to time were generally amazingly competent grad students. Plus the campus was lovely, the commute reasonable, and the benefits (like being able to take free classes) pretty cool.

I also enjoyed being something of a jack of all computer trades at Web Publishing (part of IDG) where I managed the network, the systems, the servers, the backups, the web site, and eventually acquired a very capable assistant who made the job even that much more enjoyable. And we were on the forefront of online-only publications like SunWorld and JavaWorld that provided excellent information and advice to the growing communities that used this technology.

And last, but not least, working for a couple E*Trade subsidiaries in a similar “support everything” sysadmin role. Bright creative people are almost always wonderful to work with. We worked off the Embarcadero in San Francisco and managed to have some fun together even when we were working. And, hey, taking the ferry to work was the best commute imaginable!

Some of the positions that I’ve held over the years involved having the best possible coworkers — people who were as committed as I was, who both learned from me and taught me more than I can ever thank them for. Others involved the kind of office politics that make it hard to remember that we’re supposed to be working for the same goal — to help our organizations be successful — not fighting for a position under the lime light.

you’re in a good place.
Money isn’t everything. Even living on a sailboat in the San Francisco Bay (which I did for several years) would be Heaven for some and Hell for others. Take the time to really nail down what matters to you. Is it visibility? Recognition? A sense of accomplishment? A big salary? Flexible hours? A voice in how things are done? A stake (and a say) in the outcome of your projects?

Whatever you do, don’t stop learning. Computer skills get old fast and that isn’t going to change any time soon. Spend some time every day learning something new and get your hands on some tools that might lead to the next phase in your career. Check job postings from time to time even if you have no plans to change jobs — just to keep aware of what skills are in high demand.

And put on your seatbelt. You probably can’t begin to imagine how the field is going to look in another 30 years!