Threading in Python with 3ds Max

3ds Max is a single-threaded application, which means that Python threads cannot interact with 3ds Max, either the UI or the MAXScript environment. 3ds Max is not "thread safe". This means you cannot, in a thread use any pymxs calls (MAXScript runs on the main thread).

Note:

The pymxs.mxstoken() function intended to support multi-threading is deprecated, and does not actually work.

However, threading can still be used in your scripts for accomplishing tasks that would benefit from concurrency and/or parallelism, such as file IO. Both the standard threading module and async (in Python 3) can be used.

Usefulness of multi threading

There are two main reasons to use multi threading in a program:

Making Use of Multiple Cores

The way the Python Global Interpreter Lock (GIL) works makes it impossible for two python instructions to run at the same time on the same interpreter, no matter the number of cores available. The consequence of this is that python code executing tight computation loops, no matter the number of threads at work, will only keep the equivalent of one core busy.

Heavy lifting code written in python does not benefit from multi threading, so what does? Sometimes, python programs call functions that were not implemented in python but in a different language (ex: C). The non-Python implementation of these library functions will typically release the GIL during their execution allowing threads to run normally, so that these native parts can be executed in parallel on multiple cores. So in essence, if the heavy lifting is initiated from Python but performed in another language (after the GIL has been released) the code may benefit from multiple cores. A good example of this scenario is SciPy (see details here).

It is possible for native library functions to make good use of multiple cores (and GPUS, multiple machines, or any hardware available for that matter) even if the python code that makes the calls is single threaded. If for example a python program uses TensorFlow, this library will make good use of all the available hardware to operate in an efficient way ((explanations here). It's the implementation of a given lengthy function that will distribute the work to the available hardware so that maximum parallelism can be achieved during the execution of this call.

In short, what's important to understand is that the multi-threading in python may help to make better use of cores but only under very particular circumstances, and that some library functions are able to do that anyway without any help from python threads. In all situations, it depends on the internal implementation of external libraries.

Using Threads for Concurrent IO

In most situations, performing independent IO operations concurrently takes less time than performing them sequentially, and this is the rationale for concurrent IO operations.

Reading data from devices (such as a network or hard drive) or writing data to external devices takes time but does not require a lot of work from the CPU. When possible, it is beneficial to initiate multiple IO operations at the same time and process the data as it arrives. Normal (synchronous) IO functions block the calling thread until the IO operation completes. For that reason, initiating concurrent IO operations with blocking functions requires the use of multiple threads (the main thread launches multiple threads that each perform one blocking io operation).

In 3ds Max, Python threads are useful in this scenario. As long as the worker threads don't use pymxs (or objects created with pymxs) this method can be used in 3ds Max to improve file IO performance.

Using async for Asynchronous IO

Threads can be used for achieving concurrent IO with blocking functions. Python 3 also includes the asyncio library, which allows you to do the same thing without having to pay for the memory overhead and context switching of threads. Everything is executed on the same thread through the use of an event loop (using cooperative scheduling).

The asyncio library does not require threads, but it is possible to use threads with asyncio. For example, it is possible to expose an operation running on a thread or a process as a future (see details in concurrent.futures).

In 3ds Max, it is possible to use asyncio to perform concurrent IO without creating threads or processes. In this scenario only one thread is used so the Python code can (but shouldn't) call into pymxs, because pymxs operations can themselves use blocking IO, and since these functions are not clearly identified and not awaitable, they will introduce synchronous IO to the mix.

This reduces the usefulness of asyncio, where every lengthy operation should be non-blocking.

To play well with asyncio, pymxs would need to return its blocking functions as futures, but this is not currently the case.

In python there are mechanisms that allow you to use thread-based operations or process-based operations as async operations. Calling from python threads into pymxs functions is not supported, no matter through which path this happens.