Build a Versatile OpenStack Lab with Kolla
Hone your OpenStack skills with a full deployment in a single virtual machine.
It's hard to go anywhere these days without hearing something about the urgent need to deploy on-premises cloud environments that are agile, flexible and don't cost an arm and a leg to build and maintain, but getting your hands on a real OpenStack cluster—the de facto standard—can be downright impossible.
Enter Kolla-Ansible, an official OpenStack project that allows you to deploy a complete cluster successfully—including Keystone, Cinder, Neutron, Nova, Heat and Horizon—in Docker containers on a single, beefy virtual machine. It's actually just one of an emerging group of official OpenStack projects that containerize the OpenStack control plane so users can deploy complete systems in containers and Kubernetes.
To date, for those who don't happen to have a bunch of extra servers loaded with RAM and CPU cores handy, DevStack has served as the go-to OpenStack lab environment, but it comes with some limitations. Key among those is your inability to reboot a DevStack system effectively. In fact, rebooting generally bricks your instances and renders the rest of the stack largely unusable. DevStack also limits your ability to experiment beyond core OpenStack modules, where Kolla lets you build systems that can mimic full production capabilities, make changes and pick up where you left off after a shutdown.
In this article, I explain how to deploy Kolla, starting from the initial configuration of your laptop or workstation, to configuration of your cluster, to putting your OpenStack cluster into service.Why OpenStack?
As organizations of all shapes and sizes look to speed development and deployment of mission-critical applications, many turn to public clouds like Amazon Web Services (AWS), Microsoft Azure, Google Compute Engine, RackSpace and many others. All make it easy to build the systems you and your organization need quickly. Still, these public cloud services come at a price—sometimes a steep price you only learn about at the end of a billing cycle. Anyone in your organization with a credit card can spin up servers, even ones containing proprietary data and inadequate security safeguards.
OpenStack, a community-driven open-source project with thousands of developers worldwide, offers a robust, enterprise-worthy alternative. It gives you the flexibility of public clouds in your own data center. In many ways, it's also easier to use than public clouds, particularly when OpenStack administrators properly set up networks, carve out storage and compute resources, and provide self-service capabilities to users. It also has tons of add-on capabilities to suit almost any use case you can imagine. No wonder 75% of private clouds are built using OpenStack.
The challenge remains though in getting OpenStack up and running. Even though it doesn't rely on any particular brand of hardware, it does require machines with plenty of memory and CPU cores. That alone creates a roadblock to many looking to try it. The Kolla project gets you past hurdle.What You'll Need
Kolla can be run in a single virtual machine (or bare-metal box), known as an "all-in-one" deployment. You also can set it up to use multiple VMs, which is called "multinode". In this article, I show how to deploy the former using a virtual machine deployed with KVM, the Linux virtualization service based on libvirtd. I successfully deployed Kolla on a Dell 5530 with 32GB of RAM and an i7 CPU with 12 cores, but I also did it on a machine with 16GB of RAM and four cores. You can allocate whatever you have. Obviously, the more RAM and cores, the better your OpenStack cluster will perform.
I used KVM for this deployment, but theoretically, you could use VirtualBox, VMware Desktop or another hypervisor. The base of the system is Docker, so just make sure you're using a system that can run it. Don't worry if you don't know much about Docker; Kolla uses Ansible to automate the creation of images and the containers themselves.
To install KVM, check the requirements for your distribution, keeping in mind you'll need libvirtd, qemu and virt-manager (for GUI management). On Ubuntu, this would be:
$ sudo apt-get install qemu-kvm libvirt-bin bridge-utils ↪virt-manager
On Fedora, you'd use:
$ sudo dnf -y install bridge-utils libvirt virt-install ↪qemu-kvm
On openSUSE, you'd install the KVM patterns:
$ sudo zypper -n install patterns-openSUSE-kvm_server ↪patterns-server-kvm_tools
As part of your workstation configuration, I recommend setting up bridged networking. This will enable you to connect to the Kolla VM (and the OpenStack instances you create on it) directly from the host machine. Without this, KVM defaults to a NAT configuration that isolates VMs from the host. (You'll see how to set up bridged network connections below.)
Finally, Kolla supports two Linux distributions for deployment: CentOS and Ubuntu. Your host machine can be any flavor of Linux you want (or even Windows or Mac), but the main VM will be one of the two flavors listed above. That doesn't mean you can't create OpenStack images for your OpenStack instances based on other Linux flavors. You can, and you have a lot of options. For this lab though, I'm using CentOS 7 for the main Kolla VM.Prepare Your Workstation
To work properly, Kolla wants two NICs active, and in a perfect world, these would be distinct subnets, but they don't need to be. More important for this lab is that you can access your Kolla VM and your OpenStack instances, and to do that, set up a bridge.
In my case, my workstation has two distinct networks, one internal and one external. For the internal, I used 10.128.1.0/24, but you can create a subnet that suits your needs. My subnet spans several physical and virtual servers on my lab network, including DNS servers, so I was able to take advantage of those resources automatically. Just be careful to carve out enough network resources to suit your needs. I needed only about 50 IPs, so creating a /24 was plenty for OpenStack instances and all my other servers.
You have several options on how to set up bridging depending on your Linux distribution. Most bridges can be done simply by editing config files from the command line, and others make it easy with graphical tools, like openSUSE's YaST. Regardless, the premise is the same. Instead of assigning network parameters to the physical network device—eth0, eth1, enps01 and so on—you bind the unconfigured physical device to a separate bridge device, which gets the static IP, netmask, gateway, DNS servers and other network parameters.
Historically, Ubuntu users would edit /etc/network/interfaces to set up a bridge, which might look something like this:
auto eth0 iface eth0 inet manual auto br0 iface br0 inet static address 10.128.1.10 netmask 255.255.255.0 gateway 10.128.1.1 dns-nameservers 10.128.1.2 10.128.1.3 22.214.171.124 dns-search example.com bridge_ports eth0 bridge_stp off bridge_fd 0 bridge_maxwait 0
Current versions of Ubuntu (and other distributions) use netplan, which might look something like this:
network: version: 2 renderer: networkd ethernets: enp3s0: dhcp4: no bridges: br0: dhcp4: yes interfaces: - enp3s0
See the Resources section at the end of this article for more information on using Netplan.
For distributions that use /etc/sysconfig/network/ configuration files (such as CentOS and openSUSE), a separate bridge file references a physical device. For example, ifcfg-br0 would be created along with ifcfg-eth0:
$ sudo vi /etc/sysconfig/network-scripts/ifcfg-br0: BOOTPROTO='static' BRIDGE='yes' BRIDGE_FORWARDDELAY='0' BRIDGE_PORTS='eth0' BRIDGE_STP='off' BROADCAST='10.128.1.255' ETHTOOL_OPTIONS='' IPADDR='10.128.1.10/24' STARTMODE='auto' $ sudo vi /etc/sysconfig/network/ifcfg-eth0: BOOTPROTO='none' NAME='AX88179 Gigabit Ethernet' STARTMODE='hotplug'
Depending on how your network is managed (NetworkManager, Wicked, Networkd), you should restart the service before proceeding. If things seem to be out of whack, try rebooting.Create the Kolla Virtual Machine
This deployment of OpenStack using Kolla relies on a single, beefy virtual machine. The more resources you can commit to it, the better OpenStack will perform. Here's the minimum you should assign:
- CentOS 7 (the minimal .iso is fine).
- 8GB of RAM.
- Four vCPU.
- Two NICs (can be on the same network).
- Two virtual disks (at least 40GB for the host VM OS and at least 40GB for the Cinder volumes).
This is a bare minimum. I strongly suggest at least 10GB of RAM and six vCPU. Also, if you have an SSD or NVMe drive, use that for your VM storage. Solid-state drives will improve performance dramatically and speed the initial deployment. Remember to size the disks based on your anticipated use cases. If you plan to create 200GB worth of volumes for your OpenStack instances, create a second virtual disk that's at least 200GB.
Figure 1. When creating your KVM virtual machine, remember to check the "Customize configuration before install" box, so you can add a second storage device and a second network interface.Prepare CentOS
Step through the basic configuration of CentOS and reboot. To save resources and time, don't bother installing a desktop environment. Once the system restarts, log in and perform a couple housekeeping tasks, including setting up a static IP address—no bridging here, just a static address for eth0. Don't configure the eth1 interface, but verify that it exists:
DEVICE='eth0' HWADDR='00:AA:0C:28:46:6B:91' Type=Ethernet UUID=25a7bad9-616a-40a0-ace5-52aa0af9fdb7 ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=static IPADDR=10.128.1.20 NETMASK=255.255.255.0 GATEWAY=10.128.1.1
A few times when I created the CentOS 7 VM, I found that it would rename eth0 to eth1 automatically and persist that way. Kolla requires you to specify and hard-code the interface names in the configuration file, so this unwanted name change breaks the install. If that happens, just run the following to fix it (no reboot required):
Install the Required Packages
$ sudo ip link set eth1 down $ sudo ip link set eth1 name eth0 $ sudo ip link set eth0 up
You theoretically can run the following install commands in one fell swoop, but it's better to do them individually to isolate any errors. The epel-release and other packages are required for Kolla, and if any fail, the rest of the installation will fail:
$ sudo yum update $ sudo yum install epel-release $ sudo yum install python-pip $ sudo yum install python-devel libffi-devel gcc openssl-devel ↪libselinux-python $ sudo yum install ansible git
Update pip to avoid issues later:
$ sudo pip install --upgrade pip
You'll need elements of the
kolla-ansible package for the
install, but you won't use this system version of the application to execute
the individual commands later. Keep that in mind if you run into errors during
the deployment steps:
Set Up Git and Clone the Kolla Repos
$ sudo pip install kolla-ansible --ignore-installed
The installation is done primarily from code stored in GitHub, so you'll need GitHub credentials—namely a public SSH key from your Kolla host VM added to your GitHub settings:
$ git config --global user.name "Your Name" $ git config --global user.email "[email protected]" $ git clone https://github.com/openstack/kolla $ git clone https://github.com/openstack/kolla-ansible
Figure 2. Your working directory now should look like this, containing the
kolla-ansible directories from GitHub.
Several configuration files provided by the
kolla-ansible Git repo must be
copied to locations on your Kolla host VM. The requirements.txt files checks
for all necessary packages and installs any that aren't satisfied:
Copy the Configuration Files
$ sudo cp -r /usr/share/kolla-ansible/etc_examples/kolla /etc/ $ sudo cp /usr/share/kolla-ansible/ansible/inventory/* . $ sudo pip install -r kolla/requirements.txt $ sudo pip install -r kolla-ansible/requirements.txt
Once the requirements files have run, a number of new resources will be available and must be copied to /etc/kolla/ and your working directory:
Create Cinder Volumes for LVM
$ sudo mkdir -p /etc/kolla $ sudo cp -r kolla-ansible/etc/kolla/* /etc/kolla $ sudo cp kolla-ansible/ansible/inventory/* .
It's possible to spin up your Kolla cluster without Cinder (the OpenStack
storage component), but you won't be able to create instances other than ones
built with the tiny Cirros image. Since this particular lab will use LVM for
the back end, a volume group should be created. This will be deployed on the
second virtual disk you created in your Kolla host VM. Use
vgcreate to create the volume group (to learn more, see the Cinder guide link
in the Resources section):
$ sudo pvcreate /dev/sda $ sudo vgcreate cinder-volumes /dev/sda
Figure 3. If you created a SATA disk when you set up your Kolla host VM, the
drive will show up as
Kolla gets information about your virtual environment from the main configuration file, /etc/kolla/globals.yml. Ensure that the following items are set and the lines are uncommented:
# Define the installation type config_strategy: "COPY_ALWAYS" kolla_base_distro: "centos" kolla_install_type: "binary" openstack_release: "master" # "master" ensures you're # pulling the latest release. # You also can designate specific # OpenStack versions network_interface: "eth0" # This must match the name of your # first NIC # Match first NIC on host neutron_external_interface: "eth1" # This should match the # name of your second NIC # Match second NIC on host kolla_internal_vip_address: "10.128.1.250" # Any free IP # address on your # subnet # An unused address in eth0 subnet keepalived_virtual_router_id: "51" # If initial deployment # fails to get the vip # address, change "51" # to "251" enable_cinder: "yes" enable_cinder_backend_iscsi: "yes" enable_cinder_backend_lvm: "yes" enable_heat: "yes"
Note: you can enable a wide variety of other OpenStack resources here, but for an initial deployment, I recommend this relatively minimal configuration. Also note that this configuration provides Heat and Cinder.Auto-Generate Passwords
OpenStack requires a number of different credentials, and Kolla provides a script to generate them for you. It also provides them, as necessary, to various components during deployment:
$ sudo kolla-ansible/tools/generate_passwords.py
Later, you'll need the Horizon dashboard login credentials, which are created along with the rest of the passwords. Issue the following command to get the "admin" user password:
Install the Heat Packages
$ grep keystone_admin_password /etc/kolla/passwords.yml
Heat enables ready automation of full stacks within your OpenStack environment. I recommend adding this component so you can experiment with building stacks, not just instances:
Set Up qemu as the VM Type
$ sudo pip install openstack-heat
Because you're running a nested installation of OpenStack in a virtual machine, you need to tell Kolla to use qemu as the hypervisor instead of KVM, the default. Create a new directory and a configuration file:
$ sudo mkdir -p /etc/kolla/config/nova
Create the file /etc/kolla/config/nova/nova-compute.conf and include the following:
Bootstrap the Kolla Containers
You're now ready to deploy OpenStack! If all the installation steps up to now have completed without errors, your environment is good to go.
When executing the following commands, be sure to use the version of
kolla-ansible located in the folder you downloaded from GitHub, not the
system version. The system version will not work properly.
Note that you're instructing the system to bootstrap the "all-in-one"
deployment, not "multinode". The
deploy command can take some time depending on
your system resources and whether you're using an SSD or spinning disk for
storage. Kolla is launching about 40 Docker containers, so be patient:
$ sudo kolla-ansible/tools/kolla-ansible -i all-in-one ↪bootstrap-servers $ sudo kolla-ansible/tools/kolla-ansible -i all-in-one ↪prechecks $ sudo kolla-ansible/tools/kolla-ansible -i all-in-one ↪deploy
Figure 4. Each step offers details as it's happening, so you can follow along and troubleshoot any issues.
Again, the deploy step can take some time—an hour or more. You can follow
that progress by running
sudo docker ps from a separate shell. Some
containers may appear to be "stuck" or show lots of restarts. This is normal.
Avoid any urge to halt the install.
Figure 5. Run
sudo docker ps in a separate shell to follow along as Kolla
deploys the containers it needs to build your OpenStack.
When the all-in-one deploy steps complete successfully (
you may want to make a snapshot of the VM at this point. It's a good place to
roll back to in case you run into problems later.
When the bootstrapping is complete, your OpenStack cluster will be up and running. It's actually accessible and usable in its current form, but the Kolla project provides some additional automation that adds resources and configures networking for you:
$ sudo pip install python-openstackclient --ignore-installed ↪python-glanceclient python-neutronclient $ sudo kolla-ansible/tools/kolla-ansible post-deploy
Kolla provides an initialization step that brings everything together. The
init-runonce script creates networks, keys and image flavors, among other
things. Be sure to edit the file to match your public network configuration
before proceeding. This way, your OpenStack instances will immediately have
access to your network, not the default, which won't do you any good if your
subnet doesn't match it:
$ vi kolla-ansible/tools/init-runonce
Edit the following lines to match your own network. Using the previous example network (10.128.1.0/24), your entries might look like this:
Run the Final Initialization
EXT_NET_CIDR='10.128.1.0/24' # This will become public1 EXT_NET_RANGE='start=10.128.1.100,end=10.128.1.149' # These 50 # addresses will be # floating IPs EXT_NET_GATEWAY='10.128.1.1' # Your network gateway
This is a good time to take a second snapshot of your Kolla host VM. Once you
init-runonce in the next step, you can't roll back.
Start by sourcing the admin user's openrc.sh file, and then kick off the init script:
$ source /etc/kolla/admin-openrc.sh $ kolla-ansible/tools/init-runonce
Figure 6. A sample of the output from the
If everything goes well, you now have a working OpenStack cluster. You can
access it via Horizon at the
kolla_internal_vip_address you set in the
/etc/kolla/globals.yml file (10.128.1.250 in this example):
http://kolla_internal_vip_address Username: admin Password: $ grep keystone_admin_password ↪/etc/kolla/passwords.yml
Figure 7. The OpenStack Horizon Login
After a moment, you'll be taken to the main OpenStack overview dashboard. Go ahead and explore the interface, including the Compute→Instance and Network→Network Topology. In the latter, you'll notice your public network already configured along with a private subnet and a router that connects them. Also be sure to look at the Compute→Images, where you'll see cirros, a small OS you can deploy immediately as a working instance.
Figure 8. The OpenStack Horizon Dashboard
Figure 9. Launch an instance using the provided cirros qcow2 image.
As you explore, try to keep in mind that this whole cluster is running on a single VM, and it may be slow to respond at times. Be patient, or if you can't be patient and you have more resources available, power off the cluster, and add more RAM and CPU to your virtual machine.Restarting Your Cluster
If you want to shut down your cluster, be sure there are no running processes
(like an instance in mid-launch), and just issue a
sudo poweroff command on
the Kolla host. This will shut down the Docker containers and take everything
offline. You also can issue
sudo docker stop $(docker ps -q) to stop all the
containers before shutting down. When you restart the Kolla VM, your OpenStack
cluster will take a little time to restart all the containers, but the system
will be intact with all the resources just as you left them. In most cases,
your instances won't auto-start, so you'll need to start them from the
dashboard. To restart your Kolla cluster after a shut down, you need to start
all the related OpenStack containers. Issue this command to do that:
sudo docker start $(docker ps -q)
This will find all the existing images and start them.Resources