GAR: Automating Entire OS Builds
I'm a member of the LNX-BBC Project. The LNX-BBC is a business card-sized CD-ROM containing a miniature distribution of GNU/Linux. In order to fit everything we wanted on the roughly 50MB volume, we had to build every single piece of software from scratch and weed out unnecessary files.
The first few LNX-BBC releases were done entirely by hand, pulling precompiled binaries from existing distributions and hand compiling some 200 software packages. While ultimately successful, it made the process of upgrading individual packages difficult. It was also impossible to keep our work in any sort of revision-control system.
User requests for source code revealed another problem with our development process. We had written some scripts to automate the creation of the compressed loopback filesystem and the El-Torito bootable ISO image, but the really difficult work was the compilation and installation of all the packages. We had no way of giving people a single tarball for building a BBC from scratch.
What we needed was a system for automating the compilation and installation of all of the third-party software. It needed to allow us to store our customizations in CVS and provide a simple mechanism for end users to build their own LNX-BBC ISO images.
These requirements had been noticed and met before. In 1994, Jordan Hubbard began work on the BSD Ports system. The FreeBSD operating system includes a great many programs and utilities, but it is not complete without a number of third-party programs. The BSD Ports system manages the compilation and installation of third-party software that has been ported to BSD.
Often when one asks for help with FreeBSD, an expert may answer by saying ``Just use ports!'' and listing the following commands:
cd /usr/ports/<category>/<package> make install
to suggest that the user needs to install a particular software package. This is similar to the way many Debian experts will tell people to apt-get install <package>. It's a simple way for a user to install software and related dependencies.
The Ports system is written entirely in pmake, the version of the make utility that comes with BSD. The choice to use make is both an obvious and a novel one. Make can be thought of as a language designed to automate software compilation and has many facilities for expressing build dependencies and rule-based build actions.
On the other hand, make has very limited flow control and lacks many features traditionally found in procedural programming languages. It can be rather unwieldy when used to build large projects.
As of the time of this writing, the core Ports runtime for FreeBSD has undergone 400 revisions since 1994. You can see all of the revisions and their changelog entries at http://www.freebsd.org/cgi/cvsweb.cgi/ports/Mk/bsd.port.mk. The collection of software currently contains nearly 4,000 packages.
I had made the mistake, in 1998, of making a fuss about the GNU system needing something like Ports. I made claims about how much more elegant Ports would be if written in GNU make and spent a lot of time reading the FSF's make book and the NetBSD Ports source code. It wasn't until an LNX-BBC meeting in 2001 that someone called my bluff, and I actually had to sit down and write the thing.
GAR ostensibly stands for the Gmake Autobuild Runtime because it's a library of Makefile rules that provide Ports-like functionality to individual packages. (It's actually just named GAR because that's my favorite interjection: ``Gar!'')
From the user's perspective, the GAR system may well be a tree of carefully maintained source code, ready to compile. The reality is that the system is just a tree of directories containing Makefiles, and the only thing that's stored in the GAR system itself is the information necessary to perform the steps a user would take in compiling and installing software.
The base of the GAR directory tree contains a number of directories. These directories are package categories, and within each category is a directory for each package. Inside a package directory is (among other things) a Makefile.
By way of the GAR system libraries, this Makefile provides seven basic targets for the package: fetch, checksum, extract, patch, configure, build and install.
Thus, to install Python using the BBC's GAR tree, one would cd to lang/python and run make install. To look at the source code to netcat, one would cd to net/netcat and run make extract.
Each of these seven targets runs all previous targets, though any of them may be undefined for a given package. If you run make patch, it's the same as running
make fetch checksum extract patch
fetch: this target downloads all files and patches needed to compile the package. Typically this is a single tarball, accompanied by the occasional patch file.
checksum: uses md5sum to ensure that the downloaded files match those with which the package maintainer worked.
extract: makes sure that all of the necessary source files are available in a working directory. In some cases (such as when downloading a single C source file), this will simply copy files over.
patch: if the package has to be patched (either via third-party patches or package maintainer patches), this target will perform that step.
configure: configures the package as specified in the Makefile. It will typically run the package's underlying configuration system (such as autoconf or Imake).
build: performs the actual step of compilation.
install: puts files in the proper locations and performs any necessary mop-up work.
These targets are named after their counterparts in the BSD Ports system and behave in the same manner.
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
|Non-Linux FOSS: Seashore||May 10, 2013|
|Trying to Tame the Tablet||May 08, 2013|
- Using Salt Stack and Vagrant for Drupal Development
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- New Products
- Validate an E-Mail Address with PHP, the Right Way
- Drupal Is a Framework: Why Everyone Needs to Understand This
- A Topic for Discussion - Open Source Feature-Richness?
- New Products
- New Products
- The Pari Package On Linux
- Dart: a New Web Programming Experience
- This is the easiest tutorial
2 hours 25 min ago
- Ahh, the Koolaid.
8 hours 4 min ago
- git-annex assistant
14 hours 3 min ago
- direct cable connection
14 hours 26 min ago
- Agreed on AirDroid. With my
14 hours 36 min ago
- I just learned this
14 hours 40 min ago
15 hours 10 min ago
- not living upto the mobile revolution
18 hours 2 min ago
- Deceptive Advertising and
18 hours 37 min ago
- Let\'s declare that you have
18 hours 38 min ago
Enter to Win an Adafruit Prototyping Pi Plate Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Prototyping Pi Plate Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- Next winner announced on 5-21-13!
Free Webinar: Linux Backup and Recovery
Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.
In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.