16
$\begingroup$

I am trying to run a script within Blender that utilizes multiprocessing module. According to documentation it is possible to use it ( see note at the end of "Strange errors using ‘threading’ module" ).

Pythons threads only allow co-currency and won’t speed up your scripts on multi-processor systems, the subprocess and multiprocess modules can be used with Blender and make use of multiple CPU’s too..

So I tried running below script within Blender:

from multiprocessing import Pool

def test( v ):
    print(v)
    return v

p = Pool(2)

print( p.map(test, range(5) ) )

This however spawns as many new Blender instances (with separate windows) as there are workers in p (in my case 2). Also it spews in System Console :

Can't read file: "C:\Program Files\Blender Foundation\Blender\-c", Unable to open the file.
Can't read file: "C:\Program Files\Blender Foundation\Blender\from multiprocessing.forking import main; main()", Unable to open the file.
Can't read file: "C:\Program Files\Blender Foundation\Blender\--multiprocessing-fork", Unable to open the file.
Can't read file: "C:\Program Files\Blender Foundation\Blender\2888", Unable to open the file.

I also thought about using os.fork() to do the same thing, but sadly Windows does not support that.

I've been looking all over Internet, but so far I was not able to find even one example proving that documentation is correct.

The only thing I found was thread on blenderartists.org, but it is locked and with no responses.


Update: This is a windows spesific limitation in the multiprocessing module. ~ideasman42.

$\endgroup$
  • $\begingroup$ You can find and chat with many devs by locating the following channel using an IRC application. #blendercoders freenode - Chat Room on IRC irc.netsplit.de/channels/… There is also a blenderpython channel but I forget the exact name. Someone at #blendercoders will know it. $\endgroup$ – MarcClintDion Apr 13 '14 at 4:57
  • 2
    $\begingroup$ I believe this has to do with running it in Windows. The multiprocessing module has several restrictions there, because it cannot fork, but instead spawns a new interpreter to import the module. In this case, it would seem that the arguments that would cause Python to do this don't work on Blender; the multiprocessing module must be told where to find a compatible Python interpreter instead. $\endgroup$ – Yann Vernier Jun 13 '14 at 11:52
  • $\begingroup$ few years ago there was a master-class in autodesk area that was about python multi thread I do not know if it is useful for blender but I think the concept is same. $\endgroup$ – daniel Oct 31 '14 at 16:30
  • $\begingroup$ Your code works in 2.73 . Guess they've fixed something $\endgroup$ – Highstaker Mar 10 '15 at 13:16
  • 1
    $\begingroup$ Maybe you'd like to file a bug report to get more considered answer? Errors from such a small amount of code tend to mean the problem isn't with the code.. $\endgroup$ – zeffii Jun 15 '15 at 14:52
7
$\begingroup$

This looks like a kind of ms-windows specific problem... at least a conflict with having an embedded Python interpreter and the environment which multiprocessing expects. (its not exactly a bug, in that its not like Blender devs can fix some mistake to get this working).

Currently multiprocessing makes the assumption that its running in python and not running inside an application.


I managed to get multi-processing working on ms-windows, doing some workarounds.

  • sys.executable needs to point to Python executable.
  • The scripts __file__ needs to point to a file on-disk
    (not always the case - when executing a text block for example).

Heres an example of a workaround:

import sys
# are we running inside Blender?
bpy = sys.modules.get("bpy")
if bpy is not None:
    sys.executable = bpy.app.binary_path_python
    # get the text-block's filepath
    __file__ = bpy.data.texts[__file__[1:]].filepath
del bpy, sys

# end workaround!
# --------------------------------------------------

# Example from
# https://docs.python.org/3/library/multiprocessing.html
from multiprocessing import Process
import os

def info(title):
    print(title)
    print('module name:', __name__)
    if hasattr(os, 'getppid'):  # only available on Unix
        print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(name):
    info('function f')
    print('hello', name)

def main():
    info('main line')
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()


if __name__ == '__main__':
    main()
$\endgroup$
  • $\begingroup$ I have a feeling this will not allow me to access blender specific things, so not overly useful. What I want to do is to run in parallel multiple renders of different parts of sequence/scene. I have 8 cores and Blender does not utilise all of them when rendering VSE result. Currently my solution is to open 4-6 Blender instances, change start/end frames by hand and render. Which is annoying. I would like to achieve this with one click. $\endgroup$ – elmo Jun 22 '15 at 9:23
  • $\begingroup$ Right, You can't access anything which is Blender specific (unless you compile Blender as Python module, and even then... you wont be able to pass Blender data as arguments). $\endgroup$ – ideasman42 Jun 22 '15 at 22:21
  • $\begingroup$ @elmo, From the sounds of it, you don't actually want to use Python's multiprocessing module at all. Please ask a new question ~ Blender+Python can do this, but not using multiprocessing. $\endgroup$ – ideasman42 Jun 22 '15 at 22:22
2
$\begingroup$

I entirely forgot I asked that question, but since then I've worked out a workaround for the issue.

I use Blender as video editor (even though it is not its primary purpose it works really well for my needs while still being free).

I ended up creating and add-on script that will launch multiple instances of Blender to render separate sections of my video thus utilising multiple cores in my CPU.

Full solution and actual add-on with that functionality can be seen here:

https://github.com/elmopl/ktba/blob/master/scripts/addons/parallel_render.py

It's not perfect in how it is coded, but it works fine. I simply save my current project's state into temporary .blend file and then launch multiple headless Blender instances (using subprocess module) that load that temporary project and launch into main() of my add-on. That main() is responsible for handling communication with main instance and performing render operation of headless instances.

It seems to work quite well.

Usage of addon documented here: https://github.com/elmopl/ktba/wiki/Addons#parallel-render

That solution could be split and wrapped up into separate module that does purely subprocess call + handles communication layer. Then it could be reused by others.

$\endgroup$
  • 1
    $\begingroup$ Alternatively, you can use Flamenco, which is what we use & develop at the Blender Institute. $\endgroup$ – dr. Sybren Mar 14 '18 at 14:25
1
$\begingroup$

I think this statement in the same docs says a lot:

"So far, no work has gone into making Blender’s python integration thread safe, so until its properly supported, best not make use of this. "

The statement that subprocess and multiprocess can be used with Blender isn't false, just may be interpreted in reverse with regards to multiprocess module.

  • subprocess can call external apps fine with .call or .Popen, non-blocking or blocking. Works as expected, threaded or non threaded.
  • multiprocess probably doesn't work (right now) as you might expect, how I interpreted this statement originally was, "you can call several instances of Blender from a python program using multiprocessing, that way you have control over what ends when and can use multiple CPU's.

In the end the warning is reasonable clear (I think?) that we aren't to expect threaded execution of calls to Blender functions within the one Blender executable. (for instance function calls that take a lot of time, like baking.. it is possible to write scripts to do all that and finally when all blender processes have ended , merge the result in an additional script. The work (scripting wise) may be a little bit more, but that seems to be where it's at at the moment.

Ubuntu 14.04
On Ubuntu that code leaves headless processes behind, which only finish when the main Blender executable is terminated. If there's a problem it's not entirely localized to windows.

$\endgroup$
  • $\begingroup$ Question is not about Linux. It mentions Windows explicitly (I've updated title to make it more obvious). $\endgroup$ – elmo Jun 15 '15 at 13:16
  • 2
    $\begingroup$ If you stopped reading just because I mentioned Linux then you are missing the point, the answer is pretty OS agnostic. $\endgroup$ – zeffii Jun 15 '15 at 14:07
  • 3
    $\begingroup$ Being "thread safe" has nothing to do with multiprocessing. So that part is not really relevant to question. I am looking for a solution that will allow me to write a plug-in for blender so I can just press one button to execute things in parallel (render video sequence). Quite likely I can write a script to a file and then launch it somehow in different instances of Blender, but that's not really point of that question. Answer will quite likely be very OS specific as it seems there is an issue with multiprocessing specifically on Windows. $\endgroup$ – elmo Jun 15 '15 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.