Embedding Python in Your C Programs

The language of choice for large, high-performance applications in Linux is almost always C, or somewhat less often C++. Both are powerful languages that allow you to create high-performance natively compiled programs. However, they are not languages that lend themselves to runtime flexibility. Once a C/C++ application is compiled, its code is pretty much static. At times, that can be a real hindrance. For example, if you want to allow users of a program to create plugins easily that extend the application's functionality, you have to deal with complex dynamic linking issues that can cause no end of headaches. Additionally, your users will have to know C/C++ in order to extend the application, which severely limits the number of people capable of writing extensions.

A much better solution is to provide your users with a scripting language they can use to extend your application. With a scripting language, you will tend to have much more runtime flexibility, as well as shorter development times and a lower learning curve that will extend the base of users capable of creating extensions.

Unfortunately, creating a scripting language is very much a nontrivial task that easily could become a major portion of your program. Fortunately, you don't need to create a scripting language. With Python, you can embed the interpreter directly into your application and expose the full power and flexibility of Python without adding very much code at all to your application.

Including the Python interpreter in your program is extremely simple. Python provides a single header file for including all of the definitions you need when embedding the interpreter into your application, aptly named Python.h. This contains a lot of stuff, including several of the standard headers. For compiling efficiency, it might be nice if you could include only those parts of the interface that you actually intend to use, but unfortunately Python doesn't really give you that option. If you take a look at the Python.h file, you'll see that it defines several important macros and includes a number of common headers that are required by the individual components included later in the file.

To link your application to the Python interpreter at compile time, you should run the python-config program to get a list of the linking options that should be passed to the compiler. On my system, those are:

-lpython2.3 -lm -L/usr/lib/python2.3/config

So, how much code does it take to run the Python interpreter from a C app? As it turns out, very little. In fact, if you look at Listing 1, you'll see that it can be done in as little as three lines of code, which initialize the interpreter, send it a string of Python code to execute and then shut the interpreter back down.

void exec_pycode(const char* code)
{
  Py_Initialize();
  PyRun_SimpleString(code);
  Py_Finalize();
}

Or, you could embed an interactive Python terminal in your program by calling Py_Main() instead, as in Listing 2. This brings up the interpreter just as if you'd run Python directly from the command line. Control is returned to your application after the user exits from the interpreter shell.

Embedding the interpreter in three lines of code is easy enough, but let's face it, just executing arbitrary strings of Python code inside a program is neither interesting nor all that useful. Fortunately, it's also far from the extent of what Python allows. Before I get too deep into what it can do though, let's take a look at initializing the environment that Python executes within.

When you run the Python interpreter, the main environment context is stored in the __main__ module's namespace dictionary. All functions, classes and variables that are defined globally can be found in this dictionary. When running Python interactively or on a script file, you rarely need to care about this global namespace. However, when running the embedded interpreter, you'll often need to access this dictionary to get references to functions or classes in order to call or construct them. You also may find that you occasionally want to copy the global dictionary so that different bits of code can be run in distinct environments. For instance, you might want to create a new environment for each plugin that you load.

To get at the __main__ module's dictionary, you first need to get a reference to the module. You can do this by calling the PyImport_AddModule() function, which looks up the module name you supply and returns a PyObject pointer to that object. Why a PyObject? All Python data types derive from PyObject, which makes it a handy lowest-common denominator. Therefore, almost all of the functions that you'll deal with when interacting with the Python interpreter will take or return pointers to PyObjects rather than another more specific Python data type.

Once you have the __main__ module referenced by a PyObject, you can use the PyModule_GetDict() function to get a reference to the main module's dictionary, which again is returned as a PyObject pointer. You can then pass the dictionary reference when you execute other Python commands. For example, Listing 3 shows how you could duplicate the global environment and execute two different Python files in separate environments.