Achieving Continuous Integration with Drupal

Where do you run all this stuff? The "simple" answer is to run it on the same server as Jenkins. However, Jenkins probably is not running on your production servers, so immediately your test environment is different from production. Do you know that when you install Jenkins via your distro's package manager that it does not pull in some other package that your site might end up using in testing but then fail because it is missing in production?

This points to an even deeper issue. You cannot create a clone of your production environment unless you know exactly what your production environment really is. What packages are installed? What configuration files are in place? What dæmons are running? What security updates have been installed? Running a production Web site leads to all kinds of unexpected issues and surprises, and even the best-intentioned, well-meaning sysadmins are likely to solve a crisis by changing some configuration on the server by hand. You have to make sure those changes always get propagated to your test environment. For that matter, you have to make sure they are permanently maintained in your production environment too.

This leads directly to the topic of DevOps and server configuration management. The only way to be sure your production environment is the way you expect is to automate its configuration, and the only way to ensure that your test environment is a clone of your production environment is to use exactly the same automated configuration to build it. There are good open-source tools for doing this; Puppet and Chef are the two I am familiar with. However, Puppet and Chef are programming languages in their own right. Once you go down this path, you are now maintaining two completely different pieces of software: your Web application and the infrastructure automation to run it. At this point, you need to make a recursive call to start reading this article over at the beginning, because you will need to use Continuous Integration on your infrastructure automation just like you do for your Web app. So, your Web app needs a production and test environment, all of which is running in your production infrastructure environment; now you need a test infrastructure environment in which to test updates to your infrastructure code before rolling them out to production. If you are using Jenkins to run your CI process, and Jenkins is deployed as part of the infrastructure you are developing, then...your brain just hit a stack overflow and exploded. Ooops.

To be clear, this is all doable, and there may be simplifying assumptions you can make to reduce the effort. However, if you make the mistake of thinking of your server configuration as something you can just build once and forget about, your Web site is eventually going to suffer for it.

Make all versions easily accessible. When it's time to push to production, you want to create a symbolic tag in your version control system that says what you released when. If you release frequently, you'll end up with a lot of tags, but that's okay; they're cheap. You probably will end up creating these tags in the script you create to automate deployment.

Maintain an audit trail. Your VCS gives you a commit history for your source code, but you need more than that. When something goes wrong, you easily should be able to point to the date/time/individual that played a part and quickly get the information you need. Who pushed the release to production earlier today? Who added a new domain name to Apache virtual host configuration? Can you verify that the SSH key for the employee that left last week has been removed? Most changes will be in your Web site source code, but some will be in your infrastructure configuration code, so you will want a unified view of the changes.

Automate site deployment. Okay, so you are working in small batches with frequent commits, testing every time in a clone of the production environment. Now it needs to be easy to push your new application to the live environment. If you've automated your infrastructure as described and already have a system in place for deploying new code commits to your testing environment, this should be a pretty small additional step. It has to be simple, fast and reliable; you want to be able to push a release and go have lunch five minutes later without worrying about it.

Measure results and iterate rapidly. There are many great monitoring and measurement tools available that check for things, such as error logs, page load performance, server performance, A/B testing and more. Because you've automated your infrastructure configuration, integrating these onto your servers is not that much additional work, but you still have to decide which ones to use, how best to install them, and how to get the data you need out of them most efficiently.

Use an Existing System

Whew! Okay, be honest. How likely is your company actually to make the investment to build and deploy an automated CI infrastructure as described above? I thought so. The fact is that infrastructure is not your specialty. (If it is, can we hire you?) You build exceptional Web sites, and you should not spend so much time and effort also building the servers to run it.

Your alternative is to use a system someone else built for you. Several exist, each with different properties, and with more arriving all the time. I happen to be the lead engineer for Acquia Cloud, so let me quickly demonstrate how Acquia Cloud provides everything you need to implement CI for your Web site.

Figure 1. The Workflow page is the centerpiece of Acquia Cloud's CI system.

Use a source code repository. Acquia Cloud provides both Git and SVN repositories. The URL is displayed right at the top.

Make small, frequent changes. Acquia Cloud provides a development, staging and production environment for your site. You can deploy any branch or tag from your repository in any of them. When you deploy a branch in an environment, every commit to that branch is deployed to that environment. This makes the Dev environment perfect for initial integration testing. Set it to deploy the "master" (Git) or "trunk" (SVN) branch, and every developer's commits are available immediately for initial experimentation.

Automate testing. Every time you deploy code or perform various other actions, Acquia Cloud runs "Cloud Hooks". These are simple scripts that you put into your code repository to perform any actions you want. Each hook is tied to specific actions in a specific environment—for example, all scripts in the hooks/post-code-deploy/prod directory of your repository run when you deploy code to the Production environment. The hook scripts run in sorted order until the first one fails, and the output from all hook scripts is available at the end. This is the perfect way to run your test scripts, scrub a database, perform a load test or anything else.

Test in a clone of the production environment. This is the biggest payoff with Acquia Cloud. We maintain each of these environments—Dev, Stage and Prod—for you. You can choose whether they are on the same or different servers, and whether they are redundant and load balanced or running on a single VM, but we ensure that as far as your Web application is concerned, the configuration is identical. Of course, we also provide 24/7 monitoring, backups, security updates and critical fixes—all the things you would have to do on your servers yourself.

Make all versions easily accessible. As you can see in Figure 2, you can always revert back to any specific tagged version or branch in any environment.

Figure 2. The Code selector lets you deploy any branch or prior release version to any environment.

Have an audit trail (that is, a blame list). Our task log is your audit trail. It shows code commits, but also all changes to your Web environment: domain names, SSH keys, server launches and so on. It shows you exactly what date and time each action took place, with the option to show the full detail for the command.

Figure 3. The Task log shows all changes to any of your site's environments.

Figure 4. Details are available for every task.

Automate site deployment. To deploy a release from one environment to another, simply drag and drop on the UI (or use our API or Drush CLI to do the same thing). If you drag code from an environment deploying a branch, it creates a symbolic tag at the tip of that branch and deploys the tag in the target environment. If you drag an environment deploying a tag, it just deploys the same tag in the same environment. You always can deploy any branch or any previous tag in any environment just by selecting it from the drop-down list (or, again, via our API or CLI).

Figure 5. Deploying code is a simple drag-and-drop operation (API and CLI also available).

Measure results and iterate rapidly. This article mostly been has about Acquia Cloud, but Acquia Cloud is itself just a feature of the Acquia Network that provides a wide variety of tools to improve your site, such as expert Drupal configuration advice, SEO optimization, faceted search, performance monitoring, load testing and spam blocking, plus services like education, training and support. The Acquia Network is part of every Cloud subscription and includes all of these tools, most for free.

Figure 6. The Acquia Network provides resources to understand and improve your site's results.

To view a video overview that showcases how to develop with Acquia Cloud for your Drupal site, I have a Webinar on this very topic available at http://ow.ly/cUNlL. To sign up for a completely free version of Acquia Cloud, visit http://network.acquia.com/freecloud.

______________________

Barry Jaspan is a serial software engineer and entrepreneur who has been creating and selling open-source software products literally since he was 12 years old (many moons ago!).

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thanks

Kadrush Shijaku tipps's picture

I am really impressed from your immaginative thinking and impressive stated information, i am glad.

Regarding Post

Peoples Plumber's picture

For example, if I actually have a branch known as "develop" hand-picked within the Dev atmosphere and that i push changes to that, the changes square measure mechanically deployed to the Dev servers. This push won't trigger the post-code-deploy hook, though. The hook is barely triggered if I switch to a special branch or tag. http://www.peoplesplumber.co.uk/plumber-lslington.html

I really enjoyed the quality

Truck Games's picture

I really enjoyed the quality information you offer to your visitors for this blog. I will bookmark your blog and have my friends check up here often. Truck Games

Another specialty that not

Anonymous's picture

Another specialty that not plenty of people know, is that the tremendous brand Ferrari has a range of co branded http://www.ewcoxusa.com with the Panerai collection and all of the watches in the range are available as premium Panerai watches .

To get a design and style to

Anonymous's picture

To get a design and style to survive the check out of time and be handed down from generations to generations speaks volumes of your related top quality. Swiss brand name cheap rolex watches are between the couple of objects in trend that time has not managed to erode. If something, they only get much better with time.

Thanks for sharing your vast

johnmathew's picture

Thanks for sharing your vast experience and tips for the upcoming young developers.
Drupal training in chennai

The sooner the better. . .

Joseph hamshey's picture

The sooner the better. . .

thanks

Anonymous's picture

I started to write a rebuttal from another hosting perspective and then realized I lost the battle when I saw that you could

Make all versions easily accessible. As you can see in Figure 2, you can always revert back to any specific tagged version or branch in any environment.

and although I would never use the generic word "tags" in that case what a neat GUI for this.
Thanks for this information. I'm on my way to try it out tonight.

Reply to comment | Linux Journal

SEOPressor's picture

Hi there! Do you know if they make any plugins
to protect against hackers? I'm kinda paranoid about losing
everything I've worked hard on. Any tips?

Testing tools?

brad.bulger's picture

Nice article! Any pointers - links, suggestions, etc. - about doing proper complete automated testing of Drupal sites? We need to get away from the archaic "click-around-and-try-stuff" method...

Great post and helpful

abasnad's picture

Great post and helpful information. Thank for sharing.

Jalantikus.com Download Game PC dan Android Gratis Terbaru dengan Server Lokal

What about..

Anonymous's picture

What about Backdrop. Will Acquia hosting support Backdrop?

No control over hosting

JvE's picture

The one problem I have with all this is that in Europe I find that most customers we build sites for want to either do or arrange their own hosting.
Therefore the automated deployment we use internally from dev to test cannot be reused. We pretty much have to maintain manual deployment steps for a plethora of different hosts and hosting environments.

Cloud hooks

jh3's picture

Any idea when the post-code-deploy hook will work on environments that are set to automatically deploy code from a branch when pushed to it?

For example, if I have a branch called "develop" selected in the Dev environment and I push changes to it, the changes are automatically deployed to the Dev servers. This push will not trigger the post-code-deploy hook, though. The hook is only triggered if I switch to a different branch or tag.

"Soon"

A little birdie's picture

"Soon"

Great article, Barry

rszrama's picture

Love the article, Barry. Great information and very readable. I've benefited from Acquia Cloud's infrastructure management many times over. I'm no better at infrastructure than I am at design, and I suck at design. : )

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix