Embedding Python in Your C Programs

C, meet Python. Python, this is C. With surprisingly little effort, the Python interpreter can be integrated into your program to add features quickly that could take months if written entirely in C.
Getting a Callable Function Object

Now, let's say that instead of having our expression calculator execute a list of expressions, you'd rather have it load a function f() from the Python file and execute it a variable number of times to calculate an aggregate total, based on a number provided on the command line. You could execute the function simply by running PyRun_SimpleString("f()"), but that's really not very efficient, as it requires the interpreter to parse and evaluate the string every time it's called. It would be much better if we could reference the function directly to call it.

If you recall, Python stores all globally defined functions in the global dictionary. Therefore, if you can get a reference to the global dictionary, you can extract a reference to any of the defined functions. Fortunately, the Python API provides functions for doing just that. You can see it in use by taking a look at Listing 5.

To obtain the function reference, the program first gets a reference to the main module by “importing” it using the PyImport_AddModule("__main__") function. Once it has this reference to the main module, the program uses the PyModule_GetDict() function to extract its dictionary. From there, it's simply a matter of calling PyDict_GetItemString(global_dict, "f") to extract the function from the dictionary.

Now that the program has a reference to the function, it can call it using the PyObject_CallObect() function. As you can see, this takes a pointer to the function object to call. Because the function itself already exists in the Python environment, it is already compiled. That means when you perform the call, there is no parsing and little or no compilation overhead, which means the function can be executed quite quickly.

Passing Data in Function Calls

At this point, I'm sure you're starting to think, “Gee whiz, this is great but it would be a whole lot better if I could actually pass some data to these functions I'm calling.” Well, you need wonder no longer. As it turns out, you can do exactly that. One way is through the use of that mysterious NULL value that you saw being passed to PyObject_CallObject in Listing 5. I'll talk about how that works in a bit, but first there is a much easier way to call functions with arguments that are in the form of C/C++ data types, PyObject_CallFunction(). Instead of requiring you to perform C-to-Python conversions, this handy function takes a format string and a variable number of arguments, much like the printf() family of functions.

Looking back at our calculator program, let's say you want to evaluate an expression over a range of noncontiguous values. If the expression to evaluate is defined in a function provided by the loaded Python file, you can get a reference as normal and then iterate over the range. For each value, simply call PyObject_CallFunction(expression, "i", num). The “i” string tells Python that you will be passing an integer as the only argument. If the function you were calling took two integers and a string instead, you could make the function call as PyObject_CallFunction(expression, "iis", num1, num2, string). If the function has a return value, it will be passed to you in the return value of PyObject_CallFunction(), as a PyObject pointer.

That's the easiest way to pass arguments to a Python function, but it's not actually the most flexible. Think about it for a second. What happens if you are dynamically choosing the function to call? The odds are that you're going to want the flexibility to call a variety of functions that accept different numbers and types of arguments. However, with PyObject_CallFunction(), you have to choose the number and type of the arguments at compile time, which hardly fits with the spirit of flexibility inherent in embedding a scripting language.

The solution is to use PyObject_CallObject() instead. This function allows you to pass a single tuple of Python objects instead of the variable-length list of native C data items. The downside here is that you will need to convert native C values to Python objects first, but what you lose in execution speed is made up for in flexibility. Of course, before you can pass values to your function as a Python tuple, you'll need to know how to create the tuple, which brings me to the next section.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Python Conference: PyCon 2006

Stephan Deibel's picture

For people that want to learn more about Python, especially those near Dallas Texas, I want to mention the upcoming community-organized annual Python conference, PyCon 2006.

Thanks!

Higher level interfaces

Craig Ringer's picture

The Python/C API is very low level, verbose, painful to work with, and highly error prone. With C++ code in particular it just sucks - you'll spend half your time writing const_cast("blah") to work around "interesting" APIs or writing piles of "extern C" wrapper functions, and the rest of your time writing verbose argument encoding/decoding or reference counting code. It's immensely frustraing to use plain C to write a Python object to wrap a C++ object.

Do yourself a favour, and once you've got embedding working, expose the Python interface to your program using a higher level tool. I hear SWIG is pretty good, especially for plain C code, but haven't used it myself (I work with heavily templated C++ with Qt). SIP (used to make PyQt) from Riverbank computing has its advantages also, and is good if you want your Qt app's API to integrate cleanly into PyQt. Otherwise, I'd suggest the amazing Boost::Python for C++ users, as it's ability to almost transparently wrap your C++ interfaces, mapping them to quite sensible Python semantics, is pretty impressive.

Boost::Python has the added advantage that you can write some very nice C++ code that integrates cleanly with Python. For example, you can iterate over a Python list much like a C++ list, throw exceptions between C++ and Python, build Python lists as easily as (oversimplified example):


#include <boost/python.hpp>
using namespace boost::python;


boost::python::list make_list()
{
    list pylist;
    for (int i = 0; i != 10; ++i)
        pylist.append( make_tuple( "fred", 10 ) );
    return pylist;
}

The equivalent Python/C API code is longer, filled with dangerous reference count juggling, contains a lot of manual error checking that's often ignored, and is a lot uglier.

With regards to the article above, it's good to see things like this written. I had real trouble getting started with embedding Python, and I think this is a pretty well written intro. I do take issue with one point, though, and that's duplicating the environment. Cloning the main dict does not provide separate program environments - very far from it. It only gives them different global namespaces. Interpreter-wide state changes still affect both programs. For example, if one program imports a module, the other one can see it in sys.modules ; if one program changes a setting in a module, the other one is affected. Locale settings come to mind. Most well designed modules will be fine, but you'll run into the odd one that thinks that module-wide globals are a good idea and consequently chokes.

Unfortunately, the alternative is to use sub-interpreters. Sub-interpreters are a less than well documented part of Python's API, and as they rely on thread local storage they're hopeless in single threaded programs. They can be made to work (see Scribus, for example) but it's not overly safe, and will abort if you use a Python debug build.

When you combine this with a GUI toolkit like Qt3 that only permits GUI operations from the main thread (thankfully, the limitiation is relieved by Qt4), this becomes very frustrating. If you're not stuck with this limitation, you can just spawn off a thread for your users' scripts, and should consider designing your interface that way right from the start.

Embedding Python is handy. Have fun.

P.S: LJ staff, please fix your comment engine's braindeath about leading whitespace, < and > chars in <code> sections, and blank lines in <code> sections. Thankyou. Repeated &nbsp; entities are not a fun way to format text.

--
Craig Ringer
craig@postnewspapers.com.au

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix