Improving Perl Application Performance

A number of open-source Perl packages are available. Hopefully, I could find a standard deviation calculation that was faster than my best attempt so far. I found and downloaded a statistics package from CPAN called Statistics::Descriptive. I created a function called std_dev_pm that used the Statistics::Descriptive package. The code for this function is shown in Listing 4.

sub std_dev_pm {
  my $stat = new Statistics::Descriptive::Sparse();
  $stat->add_data(@_);
  return $stat->standard_deviation();
}

Using this function, however, produced a result of 6.80 s/iter; 48% worse than the baseline std_dev function. This is not altogether unexpected considering that the Statistics::Descriptive package uses an object interface. Each calculation includes the overhead of constructing and destructing a Statistics::Descriptive::Sparse object. This is not to say that Statistics::Descriptive is a bad package. It contains a considerable number of statistical calculations written in Perl and is easy to use for calculations that don't have to be fast. However, for our specific case, speed is more important.

All languages have good and bad qualities. Perl, for example, is a good general-purpose language but is not the best for number-crunching calculations. With this in mind, I decided to rewrite the standard deviation function in C to see if it improved performance.

In the case of the data collection application, it would be counter-productive to rewrite the entire project in C. Quite a few specific Perl utilities make it the best language for most of the application. An alternative to rewriting the application is to rewrite only the functions that specifically need performance improvement. This is done by wrapping a standard deviation function written in C into a Perl module. Wrapping the C function allows us to keep the majority of the program in Perl but allows us to mix in C and C++ where appropriate.

Writing a Perl wrapper over an existing C or C++ interface requires using XS. XS is a tool that is distributed with the Perl package, and it is documented in the perlxs Perl document. You also need some of the information located in the perlguts document. Using XS, I created a Perl package called OAFastStats containing a standard deviation function implemented in C. This function, shown in Listing 5, can then be called directly from Perl. For comparison purposes, this standard deviation function will be called std_dev_OAFast.

double
std_dev(sv)
    INPUT:
        SV *            sv
    CODE:
        double sum = 0;
        double sumsq = 0;
        double mean = 0;

        /* Dereference a scalar to retrieve
           an array value */
        AV* data = (AV*)SvRV(sv);

        /* Determine the length of the array */
        I32 arrayLen = av_len(data);

        if(arrayLen > 0)
        {
          for(I32 i = 0; i <= arrayLen; i++)
            {
              /* Fetch the scalar located at i
                 from the array.*/
              SV** pvalue = av_fetch(data,i,0);

              /* Dereference the scalar into
                 a numeric value. */
              double value = SvNV(*pvalue);

              /* collect the sum and the
                 sum of squares. */
              sum += value;
              sumsq += value * value;

            }
          mean = (sum/(arrayLen+1));
          RETVAL = sqrt((sumsq/(arrayLen+1)) -
                        (mean * mean));
        }
        else
        {
          RETVAL = 0;
        }

    OUTPUT:
        RETVAL

The comparison between the baseline standard deviation function and the C function wrapped with XS is presented in Table 2, showing a significant speedup. The C function (std_dev_ref_OAFast) is 1,175% faster than the baseline function (std_dev), and it is 395% faster than the best Perl implementation (std_dev_ref_sum).