Orchestration with MCollective

I originally got into systems administration because I loved learning about computers, and I figured that was a career that always would offer me something new to learn. Now many years later that prediction has turned out to be true, and it seems like there are new things to learn all the time. In particular, every now and then a new technology comes around that dramatically changes how sysadmins do their jobs. For instance, in the October 2012 issue of LJ, I wrote an article titled "How to Deploy a Server" where I described the progression of how sysadmins deployed servers from by-hand bespoke configuration, to images, to post-install scripts and finally with configuration management.

So in this article, I'm going to expand on that concept to talk about how to use orchestration tools (in particular, MCollective) to manage orchestration tasks on servers post install. Many MCollective installation guides already exist, so I won't repeat that here; instead, my goal is to provide examples of how these tools can automate administration tasks further and to describe how I personally use them. And although I'm specifically discussing MCollective, these same concepts can be adapted and applied to any number of other orchestration tools.

These days, configuration management still is one of the most popular ways for sysadmins to configure a server, but over time, many administrators started pushing these tools past configuration management into what's being called orchestration. Orchestration refers to tools to help you push changes—in particular, software installation and updates—across your environment in a measured, staged way.

Although some administrators might be fine with pushing software updates randomly, if you want smooth upgrades, usually you want to follow an approach where you might update one server first, then if that succeeds, update a few more before updating the rest. Before you update software, you may want to notify upstream systems so they can stop sending traffic, and after you update the software, you may want to restart the service. This process is nothing new; it's just that in the past, administrators would do this by hand by logging in to machines one by one, or they would write custom scripts. With orchestration tools, you can perform these same steps from a centralized location.

The line between configuration management and orchestration is bit clearer with tools like Puppet and Chef than, say, with SaltStack or Ansible. Although Puppet and Chef can run in a masterless way, the default approach is to have clients check in to a master server periodically to see whether they comply with the central configuration and if not, to change until they do. Usually you have clients check in to the master in a somewhat randomized way or otherwise send them a trigger to apply changes.

Because tools like SaltStack and Ansible work on top of SSH, they already include an orchestration component that allows you to trigger certain kinds of changes from a central place in a staged way. Although you can make Puppet and Chef perform orchestration, many administrators who try it end up becoming frustrated and replace the tool with something else instead of realizing that those tools are very capable of what they were built to do but just not as strong at orchestration.

I personally ran into a similar kind of frustrating situation myself with Puppet when I was trying to use it to stage software updates. Puppet works great for configuration management, but wasn't ideal for orchestration in my experience. Instead of throwing away Puppet, I simply supplemented it with a very powerful tool, MCollective, that was expressly intended for orchestration and integrates well with Puppet.

MCollective lets you send out commands that query the value of particular Puppet facts, start and stop services, query and update software, and even start Puppet itself. MCollective also can restrict which servers run the command with the same facts from Facter that Puppet uses. So, for instance, you could send out a command that executes only on machines running a particular Linux distribution.


Kyle Rankin is VP of engineering operations at Final, Inc., the author of many books including Linux Hardening in Hostile Networks, DevOps Troubleshooting and The Official Ubuntu Server Book, and a columnist for Linux Journal. Follow him @kylerankin