Comparing Java Implementations for Linux

Let's start this section with a disclaimer: writing meaningful benchmark programs is very difficult, and no single benchmark can do justice to all aspects of a complex system such as a Java implementation. The benchmark I used is no exception. There are many issues it doesn't address; for example, it doesn't cover multi-threading issues, database access or graphics performance.

That said, let's look at the benchmark. The benchmark program does what most object-oriented programs do from a technical point of view: creating objects and calling methods on them. More specifically, the benchmark creates half a million very simple account objects, adds an amount to each object created, then adds up the amounts of all objects. The result of the benchmark is the elapsed time it takes to create and process all objects, and derived from that, the number of objects created and processed per millisecond.

I ran all benchmarks on a Dell Latitude CP Notebook with a 233MHz CPU and 128MB of RAM under Red Hat Linux 6.1. The code of the benchmark program is given in Listing 1.

Listing 1. Java Benchmark Source Code

As already said, this benchmark is not perfect, but I think it does give an indication of the relative performance of different Java implementations on the same platform.

Now to the results. It comes as no big surprise that a JIT is generally faster than an interpreter, and native code is even faster than a JIT. Table 2 summarizes the results.

Table 2. Java Benchmark Results

There are some observations I feel are worth mentioning. First, Java 1.2 implementations are generally faster than Java 1.1 implementations. This is a strong argument to choose Java 1.2 over Java 1.1, in addition to the much greater functionality offered by Java 1.2. Second, the fastest Java 1.2 implementation is currently the Blackdown port with the JIT enabled. The JIT provided by Borland does not speed things up, at least not in this benchmark. Third, if you have to stick with Java 1.1 (e.g., for compatibility reasons), your best options are currently Kaffe 1.0b4 and IBM's JDK 1.1.8 with the JIT enabled. Fourth, if you don't need Java bytecodes and Java 1.2, the fastest option of all is to use the Java front end of EGCS. I was not able to test the open-source version of EGCS with gcj, but I suspect the performance of it is within a few percentage points of the version sold commercially by Cygnus/Red Hat.

Finally, I ran the benchmark on the same hardware with Sun's JDK 1.2.2 on Windows NT 4.0 with service pack 5. The results are disappointing for Linux aficionados. The benchmark reports 198 objects per millisecond with the JIT enabled, and 133 objects per millisecond in interpreter mode. In other words, the Java interpreter on Windows is faster than the fastest JIT available for Linux, and the Windows JIT is faster than native code on Linux. As long as Java performance on Linux is not as good as on Windows, I don't think Linux will become the platform of choice for Java developers. Let's hope that in the near future companies such as Sun or IBM invest at least as much time in tuning Java on Linux as they do in tuning Java on Windows now.

Being a longtime C and C++ programmer, I could not resist the temptation to write a similar benchmark in C++. Listing 2 has the code of the C++ version of the benchmark.

Listing 2. C++ Benchmark Source Code

I compiled the C++ benchmark program with both the standard C++ compiler that comes with Red Hat 6.1 (EGCS 2.91.66) and the C++ compiler included in Cygnus's Codefusion 1.0 development environment. Table 3 shows the results of the C++ benchmark.

Table 3. C++ Benchmark Results

The benchmark shows that a comparable C++ program is still faster, by a factor of 3, than the fastest Java implementation (the gcj native code compiler), and even 4.3 times faster than the fastest Java JIT.

If there is any conclusion to this simple benchmark at all, it is this: I'm very much impressed by Java as a programming language (i.e., its design), but I'm not too impressed by current Java implementations on Linux or any other platform, for that matter.

Java implementations are a good example for one of the oldest laws of computing, which says that software becomes slower faster than hardware becomes faster. No doubt, Java has its place, and very often—in fact, more often than not—its advantage in programmer productivity outweighs its disadvantage in performance compared to C++. Buying a CPU which is four times faster is usually less expensive than spending two times as long to develop an application. At least, this is true for custom-developed systems. Embedded systems or mass-marketed software with hundreds of thousands of copies are an entirely different story. Saving $5 on a CPU chip in a $100 device can make all the difference and pay for increased development time.

It comes as no surprise that Java is used today mostly for custom-developed enterprise applications, where development cost and time are of paramount importance. As long as Java implementations are not within 80% of the performance of comparable C++ applications, I don't think we will see off-the-shelf products such as word processors or spreadsheets written in Java.

Wouldn't it be nice to have a programming language with the elegance and simplicity of Java and the efficiency of C? Well, maybe this will be the next big open-source project.

All listings referred to in this article are available by anonymous download in the file ftp.linuxjournal.com/pub/lj/listings/issue76/4005.tgz.