Python in the Cloud

A basic introduction to using the Python boto library to interact with AWS services and resources.

This article explores using the boto library to work with resources in the Amazon Web Services (AWS) cloud. For those wondering, the boto name refers to a species of freshwater dolphin found in the Amazon river. boto is maintained by Mitch Garnaat who works on the Eucalyptus team at Canonical. The library covers many, though not all, services offered by AWS (see the boto Web site for a current listing).

But, before I start on the use and care of boto, let me take a little diversion. It probably goes without saying, but you need to set up an AWS account if you want to play along. If you already have an Amazon account, you just need to register with Amazon at the AWS site (see Resources) to set up your AWS account. Otherwise, you need to set up an Amazon account first. As part of the setup for your AWS account, you will be issued some credentials (you will be using those later). Note, if you are a new user to AWS, check out the free tier offer (see Resources). This allows you to kick the tires at no charge for the most part. For this article, I try to stick to the offer restrictions as much as possible.

With the AWS setup out of the way, let's move on to installing boto. At the time of this writing, the current version is 2.0b4, which is what I use in this article. boto is available from the Python Package Index (PyPi), and you can install it with easy_install boto. Remember, if you want it to be a system-wide library, do the previous as a superuser. You also can go to either PyPi or the boto site and download a tarball, and then do a Python setup.py install. The PyPi site has the latest version only; the boto site has a variety of versions available.

Now that the housekeeping's done, it's time to play. As mentioned above, boto allows you to access many of the AWS services—in fact, many more than I have space to cover here. This article covers the Amazon Machine Image (AMI), Elastic Block Store (EBS), Elastic Compute Cloud (EC2), Simple Storage Service (S3) and Simple Notification Service (SNS). Where an AMI is a virtual machine, an EBS is a virtual hard drive, EC2 is the cloud you run an AMI/EBS combo in, S3 is key/object storage and SNS is a messaging system from the cloud. In all these cases, account information is needed for boto to access AWS. You'll need the AWS AccessKey and the AWS Secret Access Key. They were created when you signed up, but if you did not record them, don't fret. They are available in your AWS account page as Security Credentials.

To make things easier and more secure, there are options to include the information in the connection string. The first option is to create the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY ENV variables and set them equal to the respective keys. The other option is to create a .boto file in your home directory, and using ini style, set the keys (Listing 1).

Before going further, I should point out that AWS has a Web-based management tool for most of its services. It is called the AWS Management Console, and you can reach it by going to the AWS site and clicking on the Account tab at the top of the page (I make reference to and use this tool for some of what is to follow). The management console also is handy for confirming that the boto code actually is doing what it is supposed to be doing. Just remember to click the Refresh button at the top of the Console when in doubt.

Listing 1. boto Configuration File

#.boto file format

[Credentials]
aws_access_key_id = <some_access_key>
aws_secret_access_key = <some_secret_access_key>

What follows is a basic look at the interaction between boto and AWS. The code is kept simple both to conserve space and illustrate basic principles. To get started, I need to create an AMI of my own to work with. The easiest way to do that is to grab an existing public image and make it private. I use the Ubuntu images maintained by Canonical for my own use. The best place to find what is available is the Alestic site (see Resources). It has a handy table labeled "Ubuntu and Debian AMIs for Amazon EC2" with tabs for the AWS regions. Click on the region you want, and then select the appropriate image. For this article, I selected the Ubuntu 10.04 LTS 64-bit EBS AMI in the US-East-1 region with an AMI ID of ami-2ec83147 (the ID may be different as new images are built, see the EBS/S3 AMI Sidebar). Clicking on the arrow next to the image takes me to the AWS Management Console (after login) to start an instance from the image.

To make use of the free tier, I selected the micro instance. At this point, there is an instance of a public image running on my account. To make a private image, I could use the management console by right-clicking on the instance and selecting Create Image, but where is the fun in that? Let's use boto to do it instead (Listing 2). It's simple. Import the boto convenience function connect_ec2, and note the lack of access credentials in the connection code. In this case, they are in a .boto file in my home directory. Then, I use create_image() to create and register a private AMI using the running instance (i-c1315eaf), launched from the management console, with the name lj_test. The create_image function returns the AMI ID—in this case, ami-7eb54d17.

EBS vs. S3 AMI

Here's some enlightenment on the differences between an EBS-backed and an S3-backed AMI. S3 AMIs were the rule when AWS first started. They stored the image root device as a series of data chunks in the AWS S3 storage service. S3-backed AMIs also are referred to as instance store AMIs. Later, EBS-backed AMIs were made available. These store the root device as an EBS volume. This has some practical considerations, such as the following:

• Maximum root device size of the S3 is 10GiB and of the EBS is 1TiB.

• The boot time is faster with EBS, because the root device does not have to be assembled first.

• Stop: EBS AMI instances have the ability to be stopped, which is roughly equivalent to a paused state, in addition to being terminated. S3-backed instances only can be terminated.

For a more-detailed comparison, see the URL listed in the Resources for this article.

Security

Security, as always, is an important issue when discussing a server that is on the Internet. The standard procedures apply to EC2 instances as to regular servers. For those, see Mick Bauer's Paranoid Penguin column in previous issues of LJ, as well as any number of security references.

However, there is a special case related to AMIs that deserves mention. As demonstrated in this article, it is possible to take a publicly available AMI image and make it your own private image. This presents some special problems; see the Security URL in the Resources section. Before you create a production instance, it would be prudent to read that information and take it to heart. For the other side of the coin, consult the Alestic blog (see Resources) for how to create a secure AMI to share with others that does not leak out your private information.

Listing 2. Using boto

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import time
from boto import connect_ec2, connect_sns

# Create private image
con = connect_ec2()
con.create_image("i-c1315eaf", "lj_test")

# Launch/run instance(s)
reserv = con.run_instances("ami-7eb54d17",
    min_count=2, max_count=5, key_name='laptop',
    instance_type="t1.micro",
    placement="us-east-1d",
    disable_api_termination=True)

# Disable termination protection
con.modify_instance_attribute("i-132e457d",
    "disableApiTermination", False)

# Find running instances
res_list = con.get_all_instances(
    filters={"image-id": "ami-7eb54d17",
        "instance-state-name": "running"})

# Find instance information
for reservation in res_list:
    inst_list = reservation.instances
    for instance in inst_list:
        instance.id, instance.state
# See Figure 1 for output.


# Create a tag
con.create_tags(["i-391c2657"],
    {"Name": "lj_instance"})
con.create_tags(["vol-a9590ac2"],
    {"Name": "lj_volume"})


# Get volume
vol = con.get_all_volumes(
    filters={"tag:Name": "lj_volume"})[0]

# Create snapshot
snap = vol.create_snapshot(vol.tags["Name"]\
    + "Snap")

# Monitor snapshot creation and notify on completion
def check_snapshot(snap):
    while snap.status != "completed":
        print "Sleeping"
        time.sleep(30)
        snap.update()
    g_time = time.gmtime()
    msg_str = "Snapshot " + snap.id + "\n"
    msg_str += "of volume " + snap.volume_id + "\n"
    msg_str += "started at time "
    msg_str += snap.start_time + "\n"
    msg_str += "completed at "
    msg_str += time.asctime(g_time)
    ARN = "arn:aws:sns:us-east-1:213825411462:Lj"
    sns_con = connect_sns()
    sns_con.publish(ARN, msg_str, "Snapshot done")
    print msg_str