Sysadmin 101: Leveling Up

Senior Systems Administrator

Although some may consider people to be senior sysadmins based on a certain number of years' experience, to me, what makes someone a senior sysadmin versus a mid-level sysadmin isn't years of experience or number of places worked at, it's more a particular state of mind that one can get to via many different means. Many people get the title before they get the state of mind, and often it takes getting the title (or some of the responsibilities associated with it) to make a person level up.

The main difference between senior sysadmins and mid-level sysadmins is that one day, something clicks in senior sysadmins' minds when they realize that basically every emergency they've responded to and every project they've worked on to that point all have a common trait: given enough time and effort, they can track down the cause of just about any problem and complete just about any sysadmin task. This is a matter of true confidence, not false bravado, and it's this kind of real independence that marks senior sysadmins.

Early on in your career, certain tasks or projects just seem over your head, and you absolutely need help to complete them. Later on, you master daily tasks, but weird emergencies or complex projects still may intimidate you. Senior sysadmins have completed so many projects and responded to so many emergencies, that they eventually build the confidence such that they aren't intimidated by the next project, the next emergency or the prospect of being responsible for important mission-critical infrastructure. Like mid-level sysadmins might approach their daily tickets, senior sysadmins approach any task that comes their way.

Here are some attributes common to senior sysadmins:

  • They can perform both daily tasks and complex projects independently.

  • They understand the fundamentals behind the technologies they use and can distill complex tasks down into simple playbooks everyone on the team can follow.

  • They can be productive at a new job within a week or two.

  • Their time is spent more on large projects and odd requests that fall outside the norm.

  • They mentor other team members and have a good sense of best practices.

  • They come up with new projects and improvements and can suggest appropriate designs to solve new problems.

  • They understand their own fallibility and develop procedures to protect themselves from their own mistakes.

Again, it's the confidence and independence of senior sysadmins that separates them from mid-level sysadmins. That's not to say that senior sysadmins never ask for help. Along with the confidence of being able to tackle any sysadmin task is the humility that comes with a career full of mistakes. In fact, part of their experience will have taught them the wisdom of asking other people on the team for feedback to make sure they haven't missed anything. Often senior sysadmins will come up with multiple ways to tackle a problem, each with pros and cons, and use the rest of the team as a sounding board to help choose which approach would work best in a specific case.

Senior sysadmins' experiences exposes them to many different technologies, systems and architectures through the years. This means they start to notice which approaches work, which don't, and which work at first but cause problems in the long run. In particular, they might track some project they completed themselves through its lifetime and note how their initial solutions worked to a particular point and then either failed as it scaled, or needed to change with the advent of some new technology. The more this happens, the more senior sysadmins start to develop a natural sense of best practices and what I call the "sysadmin sense", which, like Spiderman's "spidey sense", starts to warn them when they see something that is going to result in a problem down the road, like a backup system that's never been tested or a system that has a single point of failure. It's in developing this expertise that they are able to level up to the last major level outside management.

______________________

Kyle Rankin is SVP of Security and Infrastructure at Zero, the author of many books including Linux Hardening in Hostile Networks, DevOps Troubleshooting and The Official Ubuntu Server Book, and a columnist for Linux Journal. Follow him @kylerankin