Embedding Python in Multi-Threaded C/C++ Applications

Before your threaded C program is able to make use of the Python API, it must call some initialization routines. If the interpreter library is compiled with thread support enabled (as is usually the case), you have the runtime option of enabling threads or not. Do not enable runtime threading support unless you plan on using threads. If runtime support is not enabled, Python will be able to avoid the overhead associated with mutex locking its internal data structures. If you are using Python to extend a threaded application, you will need to enable thread support when you initialize the interpreter. I recommend initializing Python from within your main thread of execution, preferably during application startup, using the following two lines of code:

// initialize Python
Py_Initialize();
// initialize thread support
PyEval_InitThreads();

Both functions return void, so there are no error codes to check. You can now assume the Python interpreter is ready to execute Python code. Py_Initialize allocates global resources used by the interpreter library. Calling PyEval_InitThreads turns on the runtime thread support. This causes Python to enable its internal mutex lock mechanism, used to serialize access to critical sections of code within the interpreter. This function also has the side effect of locking the global interpreter lock. Once the function completes, you are responsible for releasing the lock. Before releasing the lock, however, you should grab a pointer to the current PyThreadState object. You will need this later in order to create new Python threads and to shut down the interpreter properly when you are finished using Python. Use the following bit of code to do this:

PyThreadState * mainThreadState = NULL;
// save a pointer to the main PyThreadState object
mainThreadState = PyThreadState_Get();
// release the lock
PyEval_ReleaseLock();

Python requires a PyThreadState object for each thread that is executing Python code. The interpreter uses this object to manage a separate interpreter data space for each thread. In theory, this means that actions taken in one thread should not interfere with the state of another thread. For instance, if you throw an exception in one thread, the other snippets of Python code keep running as if nothing happened. You must help Python to manage per-thread data. To do this, manually create a PyThreadState object for each C thread that will execute Python code. In order to create a new PyThreadState object, you need a pre-existing PyInterpreterState object. The PyInterpreterState object holds information that is shared across all cooperating threads. When you initialized Python, it created a PyInterpreterState object and attached it to the main PyThreadState object. You can use this interpreter object to create a new PyThreadState for your own C thread. Here's some example code which does just that (ignore line wrapping):

// get the global lock
PyEval_AcquireLock();
// get a reference to the PyInterpreterState
PyInterpreterState * mainInterpreterState = mainThreadState->interp<\n>;
// create a thread state object for this thread
PyThreadState * myThreadState = PyThreadState_New(mainInterpreterState);
// free the lock
PyEval_ReleaseLock();

Now that you have created a PyThreadState object, your C thread can begin to use the Python API to execute Python scripts. You must adhere to a few simple rules when executing Python code from a C thread. First, you must hold the global interpreter lock before doing anything that alters the state of the current thread state. Second, you must load your thread-specific PyThreadState object into the interpreter before executing any Python code. Once you have satisfied these constraints, you can execute arbitrary Python code by using functions such as PyEval_SimpleString. Remember to swap out your PyThreadState object and release the global interpreter lock when done. Note the symmetry of “lock, swap, execute, swap, unlock” in the code (ignore line wrapping):

// grab the global interpreter lock
PyEval_AcquireLock();
// swap in my thread state
PyThreadState_Swap(myThreadState);
// execute some python code
PyEval_SimpleString("import sys\n");
PyEval_SimpleString("sys.stdout.write('Hello from a C thread!\n')\n");
// clear the thread state
PyThreadState_Swap(NULL);
// release our hold on the global interpreter
PyEval_ReleaseLock();