Upgrading Linux Over the Internet

The Taiwan office already had an operational gateway PC named “dragon”. Rather than upgrading it while using it to provide our connection, a second machine, “dolphin” was identified as the new gateway machine. This way, we could be sure the upgrade was successful before putting it in place, and it gave us a fall-back position in case it was not. Since the name and address of dragon were in DNS maps outside of our control, and coordination with the local ISP had proved inconvenient in the past, we had to swap the identities of the machines before proceeding.

As the new dragon would be serving both public and private networks, two network cards were installed. Simple jumper-capable NE-2000 compatible cards were chosen so that their IRQs could be easily configured. In order for our system administrator to log in to dolphin through the Internet, a minimal Caldera Linux system was installed on it. Finally, dolphin was connected to the local network.

Since the new firewall machine was no longer going to act as a mail hub for the network, an existing server running Linux, “elephant”, was nominated. Sendmail and a POP3 server were installed on elephant. Dragon was reconfigured to relay e-mail in and out of the domain rather than acting as a hub. Elephant was also configured to act as the DNS server for the internal network, with dragon as a forwarder, since elephant would no longer be directly connected to the Internet. In turn, dragon was configured to continue acting as primary DNS server for the domain to the outside world while using elephant as its resolver. This way, only publically accessible machine names and addresses would be visible from the Internet, while dragon would continue to be able to resolve all internal addresses, both public and private.

Two concerns arise when doing remote upgrades:

To avoid disruption, we decided that the upgrade should be done during the weekend in Taiwan. Since a time zone difference of exactly 12 hours exists between Massachusetts and Taipei, it was agreed that the upgrade would start on Friday at 8 PM EST, or 8 AM Saturday in Taiwan. A human would not have to be at the office in Taiwan until 9 AM when the machine was ready to be rebooted.

In advance of all this, gzipped tar files of the root, /usr and /var file systems from the Massachusetts machine were downloaded via FTP to the Taiwan office Friday night Taipei time. The exercise of downloading, building and installing SSH was also accomplished at this time.

Communication between the upgrader in the U.S. and the human sentinel in Taiwan was necessary. To avoid making expensive long-distance telephone calls (although we still ran up a $200+ telephone bill) unless it was necessary, we decided to use computer communication whenever possible. Latency eliminated e-mail as a possible choice. We chose to use talk when it worked and write otherwise.

We started by adding partitions to the disk of the target machine. Three new partitions were created with fdisk in order to hold the new root, /usr and /var file systems. Next, a reboot was needed in order to ensure the new label was in force so that new file systems could be created and the tar files restored. We used rdev to set the new root device in the kernel so that it would be ready to boot the freshly installed operating system. Then we needed to localize the machine, changing the name and address of the machine to match the Taipei office network.

Sometime in the middle of this work, it was noon in Taipei. After sending a warning note to the upgrader in the U.S. that no human would be there for an hour to restart the machine in case of a foul up, the Taiwan staff headed off to lunch.

It took two more hours after the Taipei people came back from lunch before things became almost ready. The DNS maps were copied over from dragon so the machine would be ready to step right in as primary name server for the domain.

At that point, dolphin was rebooted into the newly installed system for the first time—all seemed well. It was also almost 3 AM the next morning in Massachusetts. We were now ready to hook up the new dragon to the Internet.

The first order of business was to switch the names and IP addresses of old and new dragon before performing the physical switchover. The files /etc/hosts, /etc/hostname and /etc/init.d/network all contain references to the hostname and IP addresses that needed to be changed. Once done, the modem was unplugged from old dragon and plugged into new dragon and it was time to go for the gold.