Extending GlusterFS with Python

Although this behavior makes sense for many applications, the performance impact for many PHP applications can be severe. Without negative-lookup caching, you're likely to search half of those directories in vain before finding the one that contains each include file, every time the including page is requested. (This pattern does occur in other environments as well, including Python Web applications, but common PHP frameworks cause those applications to be hit the hardest.) Just as the effects are severe, the benefits of adding a negative-lookup cache can be significant. For example, a C version of such a translator decreased average include-search times nearly seven-fold. What could a Python version do?

Here's part of a translator based on glupy:

@lookup_fop_t
def lookup_fop (frame, this, loc, xdata):
      pargfid = uuid2str(loc.contents.pargfid)
      print "lookup FOP: %s:%s" % (pargfid, loc.contents.name)
      # Check the cache.
      if cache.has_key(pargfid) and (loc.contents.name in cache[pargfid]):
              dl.unwind_lookup(frame,0,this,-1,2,None,None,None,None)
              return 0
      key = dl.get_id(frame)
      requests[key] = (pargfid, loc.contents.name[:])
      dl.wind_lookup(frame,POINTER(xlator_t)(),loc,xdata)
      return 0

This is the function that gets called to look up a file, which is the core functionality for this example. Entry to this function represents a transition from C to Python, while its return represents a transition back to C. Calls through the "dl" object—a handle to the C dynamic library that supports glupy—also suspend the Python interpreter while they run. The Python decorator syntax allows you to hide most of the function-type details, and there's also a notable lack of type-conversion code. Most of what's there is domain-specific code, not boiler plate required by the infrastructure.

In the top half of this function, you simply check the cache to see if you already know the requested file won't be there. If the cache check succeeds, the lookup fails immediately, and you "unwind" the translator stack to report that fact. As with the registration functions, each operation type has its own specific wind (call downward) and unwind (return upward) functions as well. This represents a temporary return from the "Python world" to the "C world", and it's worth noting that these transitions between worlds might occur seamlessly many times while processing a single request. In particular, a common GlusterFS translator idiom is for a completion callback on one request to initiate the next, and if that request completes immediately (as done here), then you can have multiple requests and completions all on the stack at once.

Returning to the code, if you do not find an entry in the cache (and you already know it must not be in the standard positive-lookup cache or else you wouldn't even have been called), you pass the request on to the next translator using wind_lookup. When that next translator is done, it returns control (through the glupy meta-translator) to lookup_cbk. Here you retrieve your request context, conveniently stashed in a dictionary for you by lookup_fop, and use it to update the cache according to whether the file was found.

There are a few other less relevant details of how this particular glupy translator works, but that really is the meat of it. With less than a hundred lines of Python code, including comments and empty lines, you can add a significant piece of functionality to a real filesystem. But, how well does it really work? As it turns out, it works very well; see Table 1. A simple test reveals that the result is slower than the C-based version of the same thing, but still more than four times as fast as the baseline. Clearly, the fact that you're caching these results matters more than what language you're using to do it.

As promising as these results are, they're more of a beginning than an end. Glupy is still a very young project, and much remains to be done. Support needs to be added for a few dozen more operation types and several data structures. There still are more ways that GlusterFS calls into translators and utility functions that translators themselves call. There are many ways the glupy interface could be made more convenient, and there are undoubtedly performance or concurrency issues still to be resolved. The most important thing is that the basic infrastructure for doing all of these things already exists, and not just for GlusterFS translators. If even a highly multithreaded and asynchronous program like this can take advantage of all that Python has to offer, so can just about any other program. Thanks to Python's extension/embedding interface and ctypes module, a "best of both worlds" approach to developing complex software is more achievable than most people think.

Resources

Jython: http://www.jython.org

Cython: http://www.cython.org

GlusterFS: http://www.gluster.org

Glupy Source Repository: https://github.com/jdarcy/glupy

Negative-Lookup Caching Translator in C: https://github.com/jdarcy/negative-lookup

Zend (PHP) Framework on Include Files: http://framework.zend.com/manual/1.12/en/performance.classloading.html