Transactions and Rollback with RPM

When an upgrade works, you get a few more features or better performance. When an upgrade fails, you're in for a weekend of pain. Now, here's how to back off to the old version and keep the system up.

How many times have you installed a great new piece of software only to find you really didn't want to install it after all? To make matters worse, when you installed this software, you had to upgrade several other software packages and install additional ones from scratch. To put things back the way they were, you had to locate earlier versions of the upgraded packages from a multitude of sources, downgrade to these versions and remove any newly installed packages. Of course, if you did not keep a good record of what packages actually were changed and what their previous versions were, things got even worse. Wouldn't it be great if you instead could push one button or run a single command and roll back this upgrade?

In some environments the ability to roll back an upgrade quickly is not only desirable, it is a requirement. For instance, when upgrading a telecommunications company's equipment, software and hardware vendors are required to upgrade equipment in a limited time frame, known as a maintenance window. In this same maintenance window, they also must be able to back out any changes made by the upgrade. Failure to back out an upgrade within the maintenance window results in strict financial penalties.

Rolling Back an Upgrade the Hard Way

As desirable as an automated rollback of RPMs is, RPM has not supported this option until recently. To be fair, RPM has supported downgrading a set of packages. For instance, if you upgraded some RPM foo-1-1 to the version foo-1-2, you could use the --oldpackage switch with the rpm command to downgrade to a previous version; like this:

# rpm -Uvh --oldpackage foo-1-1.i386.rpm
Preparing...                ################# [100%]
Upgrading...
   1:foo                    ################# [100%]

If the upgrade to foo-1-2 did not require you to upgrade or install any additional RPMs, the --oldpackage switch worked fine. All you had to do was find the original foo-1-1 RPM, and you were home free. If, on the other hand, you did need to install or upgrade other RPMs on which foo-1-2 depended, you then had to search for those RPMs in various locations—on your install media, on your distribution's errata site, in various RPM repositories or on various project Web sites.

Once you had hunted down all the dependent RPMs, you would need to downgrade all the ones you had upgraded and erase the fresh ones you had installed. If, instead, you reversed this order and erased the fresh RPMs you had installed before you downgraded the RPMs that had been upgraded, you were greeted with errors from RPM complaining that these packages were required by foo-1-2. In short, the old way of rolling back a set of RPMs was painful and fraught with error.

Transactional Rollbacks to the Rescue

Early in 2002, Jeff Johnson, the current maintainer of RPM, began to remedy this rollback problem when he included the transactional rollback feature into the 4.0.3 release of RPM. This feature brought with it the promise of an automated downgrade of a set of RPMs. Like many new features, it was rough around the edges and completely undocumented, except for a few e-mails on the RPM mailing list (rpm-list@redhat.com). Over the past year and a half, transactional rollbacks have matured steadily. In the current RPM 4.2 release, which comes with Red Hat 9, transactional rollbacks are quite usable.

How RPM Transactional Rollbacks Work

Under the hood, RPM treats any set of RPMs it installs as a discrete transaction. This is true when installing one RPM by itself (a transaction of one RPM) or several RPMs simultaneously. Each of these transactions is given a unique transaction ID (TID). As each RPM is installed or upgraded, its entry in the RPM database is marked with the TID of the transaction within which it was installed. This allows RPM to track within which transaction each RPM was installed or upgraded.

To roll back an RPM transaction set, RPM must have access to the set of RPMs that were on the system at the time the transaction occurred. It solves this problem by repackaging each RPM before it is erased and storing these repackaged packages in the repackage directory (by default, /var/spool/repackage). Repackaged packages contain all the files owned by the RPM as they existed on the system at the time of erasure, the header of the old RPM and all the scriptlets that came with the old RPM.

You may be wondering how this design helps with upgrades. After all, if you upgrade an RPM you're not erasing it. You are erasing it, though, because upgrading an RPM has two parts: the new package is installed, and the old package is erased. This means every time you upgrade a set of packages, RPM first repackages all packages being updated, then installs all the new packages and, finally, erases all the old packages. When RPM repackages the old packages, it also marks the repackaged packages with the TID of the running transaction. The end result is you don't have to scour the Net, media or backups for the RPMs you updated. Because the repackaged packages contain the files that were currently on your system at the time of the upgrade, the need to restore configuration files from backup is eliminated. As a side effect, the md5 checksums of the files in a repackaged package are likely to be wrong, because RPM does not recalculate each checksum when creating the repackaged package. This is not a problem for RPM when it rolls back transactions, but you need to use the --nodigest option to manipulate repackaged packages directly.

Once the repackage directory is populated, RPM requires only a rollback target (the date to which it is rolling back) to perform the rollback. RPM then determines by TID which transactions have been applied to your system since the rollback target date. Next RPM takes this set of transactions, sorts them in the order of most recent to least recent and does the following for each one:

  • Finds all the repackaged packages that are marked with this TID.

  • Finds all the currently installed packages that are marked with this TID but do not have corresponding repackaged packages.

  • Builds a rollback transaction. Repackaged packages are added to this transaction as install elements, and installed packages that have no corresponding repackaged package are added as erase elements.

  • Runs the newly built rollback transaction.

By repeating these actions for each transaction from the most recent one to the one nearest or equal to the target date, RPM walks through all transactions that have occurred since the rollback goal and undoes them.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Empty rpms when I use rpm -Uvh --test with transational rollback

Bryan's picture

When I have repackage_all_erasures on and then do a rpm -Uvh --test it creates empty .rpm files in the /var/spool/repackage area. Has anyone else noticed this? Not sure if this is a bug or a feature.
I just checked whether this affects what happens when I first do a rpm -Uvh --test foo.rpm and then actually do a rpm -Uvh. It doesn't seem to affect anything, the empty foo.rpm is replaced by the real foo.rpm in /var/spool/repackage.

Re: Transactions and Rollback with RPM

Anonymous's picture

Watch out for the situation where you already have rollback rpms in the repackage directory with the same name.
ie. upgrade prog-1.0-1 to 1.1-1 and create a prog-1.0-1.i386.rpm file in the rollback package directory. Perform an up2date --undo, which backs out version 1.1 and puts version 1.0 back. It leaves the rollback RPMS hanging around and if you don't delete them, your next upgrade of prog-1.0 will fail to generate the rollback packages with an error like.
--------------------------------------------------------------------
Warning: The following packages are being updated, but these packages
have already been repackaged and this is currently not supported.
Repackage support for this transaction has been disabled.
--------------------------------------------------------------------
And now you have no rollback available, but the previous rollback packages have been removed!!
So remember, manually remove your rollback packages after you have performed a rollback.

This is fixed, but I forget in which version...

James Olin Oden's picture

I fixed this in the HEAD of rpm about a your ago. That said, I don't know which official versions have this fix. Basically, the rollback needed to remove the repackaged packages that it used in the rollback. If your current version of RPM does not do this, I can probably produce a patch pretty quickly.

Finally, just so you know, the latest version of RPM will have several fixes for transactional rollbacks (all done by Jeff Johnson), and will have the latest autorollback patches in it. This will be the 4.4.3 release of rpm. There are many other fixes and features.

Cheers...james

Re: Transactions and Rollback with RPM

Pedro Algarvio's picture

Couldn't this rollback feature automaticly remove the rolled back package?

Anyone, a patch?

I would do it if I could....

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix