Embedding Python in Multi-Threaded C/C++ Applications
When you embed Python within your application, it is often desirable to provide a small module that exposes an API related to your application so that scripts executing within the embedded interpreter have a way to call back into the application. This is done by providing your own Python module, written in C, and is exactly the same as writing normal Python modules. The only difference is your module will function properly only within the embedded interpreter.
Extending Python requires some understanding of how the Python interpreter manipulates objects from C. All function arguments and return values are pointers to PyObject structures, which are the C representation of real Python objects. You can make use of various function calls to manipulate PyObjects. Listing 2 is a simple example of a Python module extension written in C. This is the source to the Python crypt module, which provides one-way hashing used in password authentication.
All C implementations of Python-callable functions take two arguments of type PyObject. The first argument is always “self”, the object whose method is being called (similar to the infamous “this” pointer in C++). The second object contains all the arguments to the function. PyArg_Parse is used to extract values from a PyObject containing function arguments. You do this by passing, in the PyObject which contains the values, a format string which represents the data types you expect to be there, and one or more pointers to data types to be filled in with values from the PyObject. In Listing 2, the function takes two strings, represented by "(ss)". PyArg_Parse is similar to the C function sscanf, except it operates on a PyObject rather than a character buffer. In order to return a string value from the function, call PyString_FromString. This helper function takes a char* value and converts it into a PyObject.
C programs can easily create new threads of execution. Under Linux, this is most commonly done using the POSIX Threads (pthreads) API and the function call pthread_create. For an overview of how to use pthreads, see “POSIX Thread Libraries” by Felix Garcia and Javier Fernandez at http://www.linuxjournal.com/lj-issues/issue70/3184.html in the “Strictly On-line” section of LJ, February 2000. In order to support multi-threading, Python uses a mutex to serialize access to its internal data structures. I will refer to this mutex as the “global interpreter lock”. Before a given thread can make use of the Python C API, it must hold the global interpreter lock. This avoids race conditions that could lead to corruption of the interpreter state.
The act of locking and releasing this mutex is abstracted by the Python functions PyEval_AcquireLock and PyEval_ReleaseLock. After calling PyEval_AcquireLock, you can safely assume your thread holds the lock; all other cooperating threads are either blocked or executing code unrelated to the internals of the Python interpreter, and you may now call arbitrary Python functions. Once acquiring the lock, however, you must be certain to release it later by calling PyEval_ReleaseLock. Failure to do so will cause a thread deadlock and freeze all other Python threads.
To complicate matters further, each thread running Python maintains its own state information. This thread-specific data is stored in an object called PyThreadState. When calling Python API functions from C in a multi-threaded application, you must maintain your own PyThreadState objects in order to safely execute concurrent Python code.
If you are experienced in developing threaded applications, you might find the idea of a global interpreter lock rather unpleasant. Well, it's not as bad as it first appears. While Python is interpreting scripts, it periodically yields control to other threads by swapping out the current PyThreadState object and releasing the global interpreter lock. Threads previously blocked while attempting to lock the global interpreter lock will now be able to run. At some point, the original thread will regain control of the global interpreter lock and swap itself back in.
This means when you call PyEval_SimpleString, you are faced with the unavoidable side effect that other threads will have a chance to execute, even though you hold the global interpreter lock. In addition, making calls to Python modules written in C (including many of the built-in modules) opens the possibility of yielding control to other threads. For this reason, two C threads that execute computationally intensive Python scripts will indeed appear to share CPU time and run concurrently. The downside is that, due to the existence of the global interpreter lock, Python cannot fully utilize CPUs on multi-processor machines using threads.
- Transitioning to Python 3
- Red Hat OpenStack Platform
- Stepping into Science
- Tech Tip: Really Simple HTTP Server with Python
- Linux Journal December 2016
- CORSAIR's Carbide Air 740
- The Tiny Internet Project, Part II
- Radio Free Linux
- A Better Raspberry Pi Streaming Solution
- FutureVault Inc.'s FutureVault