Sysadmin 101: Ticketing

Tickets Distribute the Work

If you have only one sysadmin, distributing tickets and projects is easy. Once your team grows though, it's important to distribute the work so no member of the team gets burned out. Coworkers have a tendency of finding that senior member of your team who is most productive and going to them directly when they have any issue. Of course, that team member is probably already working on plenty of other tasks or may be trying to focus on an important project.

When a task is captured in a ticket, the team lead or manager can assign and reassign tickets to different members of the team to make sure no one gets burned out, and also to ensure that everyone learns how to do things. Otherwise, you end up cultivating specialists within the team that always take tickets related to certain systems, which leads to problems later when that team member goes on vacation.

Tickets Provide an Audit Trail for Changes

Every time you change a system, you create an opportunity for something to break. If you are lucky, things break immediately after you make the change. More often, you'll find that it takes some time for a change to cause a problem. You'll discover two weeks later that something stopped working, and with a ticketing system, you can pull up all of the tasks that were worked on around that time. This makes it much easier to pinpoint potential causes of a problem.

Tickets also provide an audit trail for tasks that require approval or proof of completion, like creating or revoking accounts, granting new privileges or patching software. When someone asks who said it was okay for Bob to get access to production and when it happened, you can answer the question. If you need to prove that you applied a security patch, you can point to command output that you capture and then store in the corresponding ticket.

Qualities of an Effective Ticketing System

Many different ticketing systems exist, and sometimes when you hear people complain about tickets, what they are really complaining about is a bad ticketing system. When choosing between ticketing systems, you should look for a few things.

Some systems that developers use to track code through the development process result in very complicated workflows. For a sysadmin though, the simpler the ticketing system the better. Because you already are asking a sysadmin to take time out of solving a problem to document it in a ticket, it helps if the ticketing process is fast and simple. I prefer very simple ticket workflows for sysadmins where there may be only a few states: open, assigned, in progress, resolved and closed. (I'll talk more about how I treat each of those states in the next section.)

The fewer required fields in a ticket, the better. If you want to add extra fields for tags or other information, that's fine, just don't make those fields mandatory. The goal here is to allow sysadmins to create tickets based on someone walking up and tapping them on the shoulder in less than a minute.

Ideally, the ticketing system would allow you some other way to generate tickets from a script, either from sending an email to a special address or via an exposed API. If it has an API that lets you change ticket state or add comments, all the better, as you potentially can integrate those into your other automation scripts. For instance, I've created a production deployment script that integrates with my ticketing system, so that it reads the manifest of packages it should install from the ticket itself and then outputs all of the results from the deployment as comments in the ticket. It's a great way to enforce a best practice of documenting each of your software releases, but it does it in a way that makes it the path of least resistance.

Favor ticketing systems that allow you to create dependencies or other links between tickets. It's useful to know that task A depends on task B, and so you must complete task B first. These kinds of ticketing systems also make it easier to build a master ticket to track a project and then break that large project down into individual tickets that describe manageable tasks. These kinds of systems often show all of the subordinate tickets in the master ticket, so a quick glance at the master ticket can give you a clue about where you are in a project.

______________________

Kyle Rankin is senior security and infrastructure architect, the author of many books including Linux Hardening in Hostile Networks, DevOps Troubleshooting and The Official Ubuntu Server Book, and a columnist for Linux Journal. Follow him @kylerankin