Embedding Python in Your C Programs

C, meet Python. Python, this is C. With surprisingly little effort, the Python interpreter can be integrated into your program to add features quickly that could take months if written entirely in C.
Converting Between Python and C Data Types

Python data structures are returned from and passed to the Python interpreter in the form of PyObjects. To get to a specific type, you need to perform a cast to the correct type. For instance, you can get to a PyIntObject pointer by casting a PyObject pointer. If you don't know for sure what the variable's type is, though, blindly performing a cast could have disastrous results. In such a case, you can call one of the many Check() functions to see if an object is indeed of an appropriate type, such as the PyFloat_Check() function that returns true if the object could indeed be cast to a float. In other words, it returns true if the object is a float or a subtype of a float. If you'd rather know whether the object is exactly a float, not a subclass, you can use PyFloat_CheckExact().

The opaque PyObject structure isn't actually useful to a C program though. In order to access Python data in your program, you'll need to use a variety of conversion functions that will return a native C type. For example, if you want to convert a PyObject to a long int, you can run PyInt_AsLong(). PyInt_AsLong is a safe function, and will perform a checked casting to PyIntObject before extracting the long int value. If you know for sure that the value you're converting is indeed an int, it may be wasteful to perform the extra checking—especially if it's inside of a tight loop.

Often, Python functions ask for or return Python sequence objects, such as tuples or lists. These objects don't have directly corresponding types in C, but Python provides functions that allow you to build them from C data types. As an example, let's take a look at building a tuple since you'll need to be able to do that to call a function using PyObject_CallObject().

The first step to creating a new tuple is to construct an empty tuple with PyTuple_New(), which takes the length of the tuple and returns a PyObject pointer to a new tuple. You can then use PyTuple_SetItem to set the values of the tuple items, passing each value as a PyObject pointer.

Conclusion

You should now have enough to get started with embedding Python scripts inside your own applications. For more information, take a look at the Python documentation. “Extending and Embedding the Python Interpreter” goes into more detail on going the other direction and embedding C functions inside Python. The “Python/C API Reference Manual” also has detailed reference documentation on all of the functions available for embedding Python in your program. The Linux Journal archives also contain an excellent article from Ivan Pulleyn that discusses issues for multithreaded programs that embed Python.

Resources for this article: /article/8714.

William Nagel is the Chief Software Engineer for Stage Logic, LLC, a small software development company, where he develops real-time systems based on Linux. He is also the author of “Subversion Version Control: Using the Subversion Version Control System in Development Projects”.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Python Conference: PyCon 2006

Stephan Deibel's picture

For people that want to learn more about Python, especially those near Dallas Texas, I want to mention the upcoming community-organized annual Python conference, PyCon 2006.

Thanks!

Higher level interfaces

Craig Ringer's picture

The Python/C API is very low level, verbose, painful to work with, and highly error prone. With C++ code in particular it just sucks - you'll spend half your time writing const_cast("blah") to work around "interesting" APIs or writing piles of "extern C" wrapper functions, and the rest of your time writing verbose argument encoding/decoding or reference counting code. It's immensely frustraing to use plain C to write a Python object to wrap a C++ object.

Do yourself a favour, and once you've got embedding working, expose the Python interface to your program using a higher level tool. I hear SWIG is pretty good, especially for plain C code, but haven't used it myself (I work with heavily templated C++ with Qt). SIP (used to make PyQt) from Riverbank computing has its advantages also, and is good if you want your Qt app's API to integrate cleanly into PyQt. Otherwise, I'd suggest the amazing Boost::Python for C++ users, as it's ability to almost transparently wrap your C++ interfaces, mapping them to quite sensible Python semantics, is pretty impressive.

Boost::Python has the added advantage that you can write some very nice C++ code that integrates cleanly with Python. For example, you can iterate over a Python list much like a C++ list, throw exceptions between C++ and Python, build Python lists as easily as (oversimplified example):


#include <boost/python.hpp>
using namespace boost::python;


boost::python::list make_list()
{
    list pylist;
    for (int i = 0; i != 10; ++i)
        pylist.append( make_tuple( "fred", 10 ) );
    return pylist;
}

The equivalent Python/C API code is longer, filled with dangerous reference count juggling, contains a lot of manual error checking that's often ignored, and is a lot uglier.

With regards to the article above, it's good to see things like this written. I had real trouble getting started with embedding Python, and I think this is a pretty well written intro. I do take issue with one point, though, and that's duplicating the environment. Cloning the main dict does not provide separate program environments - very far from it. It only gives them different global namespaces. Interpreter-wide state changes still affect both programs. For example, if one program imports a module, the other one can see it in sys.modules ; if one program changes a setting in a module, the other one is affected. Locale settings come to mind. Most well designed modules will be fine, but you'll run into the odd one that thinks that module-wide globals are a good idea and consequently chokes.

Unfortunately, the alternative is to use sub-interpreters. Sub-interpreters are a less than well documented part of Python's API, and as they rely on thread local storage they're hopeless in single threaded programs. They can be made to work (see Scribus, for example) but it's not overly safe, and will abort if you use a Python debug build.

When you combine this with a GUI toolkit like Qt3 that only permits GUI operations from the main thread (thankfully, the limitiation is relieved by Qt4), this becomes very frustrating. If you're not stuck with this limitation, you can just spawn off a thread for your users' scripts, and should consider designing your interface that way right from the start.

Embedding Python is handy. Have fun.

P.S: LJ staff, please fix your comment engine's braindeath about leading whitespace, < and > chars in <code> sections, and blank lines in <code> sections. Thankyou. Repeated &nbsp; entities are not a fun way to format text.

--
Craig Ringer
craig@postnewspapers.com.au

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState