Security Exercises

Regular security exercises are, bar none, the most powerful, cost-effective tool for maturing a project's information security operations—when done well. Unfortunately, courses and certifications on InfoSec tend to focus on how to implement specific controls or how to select some baseline best practices when starting from scratch. Little to no attention tends to be paid on how to test what you have and iterate on it. Prepare for a crash course.

[Note: tomorrow we will post a follow-up to this article with several example security exercises you can try with your team.]

Security Exercise? What's That?

A security exercise is a drill designed to propel a team or teams through the steps they would take in the case of a real or suspected information security problem in their organization or project. For example:

  • Tell your ops team that the server hosting your internal bug tracker has experienced data loss due to a critical RAID controller failure. Have them rebuild the server from backups on spare hardware to show that the backups are viable, spare hardware available and the process known and workable.

  • Start running an otherwise innocuous, but memory-intensive, piece of unauthorized software on a development server. See how long it takes for someone to notice and what he or she does about it.

Isn't This Dangerous?

Security exercises are not the first step in running an InfoSec program for a project of any size. The first step is coming up with a plan or set of policies appropriate to the size and complexity of the project. For a very small, all-volunteer open-source project, this may be as simple as "Our project manager, $name, accepts risk on behalf of the project and our information security officer, $name, is in charge in the case of a suspected security incident; the integrity of our code base will be prioritized first, confidentiality of yet-undisclosed vulnerability information second and availability of services third." For a larger, more complex organization with paid staff, this normally will start with a Master Information Security Policy and Procedures document supported by a number of other policy documents.

In either case, step one is establishing roles and responsibilities; step two is establishing operational expectations, and step three is testing that your policies, procedures and expectations work. If you aren't testing, you don't really know that it works.

Scheduling exercises at a predictable time and reminding others when it will happen prevents confusion among staff. It is wise to begin with low-impact exercises (more on this below) that don't leverage production systems, and move on to higher-potential-impact exercises only when the organization's infrastructure and personnel have had most of the bugs shaken out. If something as small as a runaway process on a single server can seriously impact your business, it's better to find out at a planned time with all hands on deck than at 4am on a holiday when no one who knows what to do can be reached. The whole point of security exercises is to increase resilience: raise the threshold of what is normal for your team to deal with, what your systems can shrug off.

Why Are Security Exercises Important?

When I respond to a security incident that's gone disproportionately bad—that is, far worse than the incident should have gone given the resources and security needs of the organization—it tends to be true that more than one thing has gone wrong. However, the root cause of how those things were all allowed to go wrong at once is almost one or both of two things: lack of interest in and support for information security from organization leadership, or the failure mode I call "death by supposition".

"Death by supposition" is when we make decisions based on "facts" that are supposed to be true, but have not been tested by us. For example, suppose that hardware or software will behave the way the vendor said it would. Suppose that anybody in the company remembers the incident response plan that was written, approved and stuck in a drawer two years ago. Suppose that the "best practices" written for companies in your sector don't overlook some way in which the sector has changed, or your company is unique. Suppose that your backups work the way they were designed to, and nothing has gone awry in the 25 updates since the system was put in place 18 months ago. Suppose that your VPN hardware fails closed the way the vendor's sales staff insisted that it would.

Supposition kills, and it's an insidious killer because, unlike bad leadership, it's easy to miss. We often aren't aware of our assumptions until something goes horribly wrong—better for that something to be a simulated attack than a real one, leading only to simulated consequences.

Security exercises, done right, will do the following:

  • Reveal whether systems and technical controls (still) work as expected.

  • Ensure that security, ops, leadership and other team members are on the same page.

  • Reveal holes in procedures and policies.

  • Provide your team with vital practice at operations that may someday need to be done quickly and/or under stress, especially disaster recovery and incident response procedures.

  • Provide your team with stress inoculation. This is something that SWAT teams, martial artists, search-and-rescue teams, firefighters, military and so on already know is an essential part of their live drills: getting used to something so it doesn't register as such a large stressor any more.

  • Provide non-security personnel and security personnel alike with valuable hands-on security training.

  • Improve the relationships needed to make security improvements and incident response go more smoothly.

Most important, well-executed security exercises take your organization from the land of supposition to actually knowing where your weaknesses are, where your resources should be going, and what you are doing right. Don't guess. Know.


Susan Sons' passion for education has driven her open-source efforts with Debian Edu, Edubuntu and her own initiative Frog and Owl, which helps technologists connect with educators to build more useful educational tools.