How to monitor render process in a thread-safe manner?

Question

I'm currently writing an addon, the purpose of which requires it to render several parts of active scene sequentially, preferably with UI-responsiveness intact. The problem is the UI freezes if I just directly call bpy.ops.render.render(animation=True). I get UI-responsiveness by changing the execution context to "INVOKE_DEFAULT", but then the code won't wait until a render process finished before executing the next one.

Using threading Module

Looking for an alternative, I then use Python's threading.Thread object to encapsulate render function, like this:

def _init_render_thread(self):
    self._render_thread = threading.Thread(
        target=bpy.ops.render.render, kwargs={'animation':True})

... and check the thread object's is_alive method at interval set by a Timer object. Unfortunately it's prone to occasional random crashing, as any thread-related bugs are, so I'm still stuck with this method.

Chaining Handlers

Based on jesterKing's answer, I then chain the render execution using a scene-global property and render handlers with the following code. By chaining, the operator no longer needs to be modal. It's finishing early, then leaving completion to these handlers:

@persistent
def render_pre_handler(dummy):
    props = bpy.context.scene.oha_layout_tools

    if props.render_marker_infos:
        rmi = props.render_marker_infos.pop(0)
        marker_scene_settings(bpy.context, rmi)

@persistent
def render_complete_handler(dummy):
    props = bpy.context.scene.oha_layout_tools

    if props.render_marker_infos:
        rmi = props.render_marker_infos.pop(0)
        marker_scene_settings(bpy.context, rmi)

        bpy.ops.render.render('INVOKE_DEFAULT', animation=True)
    else:
        bpy.app.handlers.render_pre.remove(render_pre_handler)
        bpy.app.handlers.render_complete.remove(render_complete_handler)
        props.marker_infos.clear()
        props.render_marker_infos.clear()

It still doesn't work. The render_complete handler promptly executes even while the render thread itself is still in progress.

Is there any thread-safe way to monitor render process, other than spawning new Blender process?

Answer 1

You can add handlers for render complete process: http://www.blender.org/documentation/blender_python_api_2_67b_release/bpy.app.handlers.html#bpy.app.handlers.render_complete

Use that to flip a flag that you can check in your main code that can check when the current render is ready before starting a new one.

Check also render_post (and render_pre) for usage in your add-on.

Answer 2

Your handlers look fine, but you definitely need to look into other ways to run multiple processes than using the threading module. Python's threading module is not fully compatible with how Blender operates and is a guaranteed way to generate lots of ugly errors farther down the road.

Other options you should consider:

• Blender's Modal Operators: These are not the same as the threading module, but are instead essentially Blender's own implementation of threading. Look at the Operator Modal template file in Blender's Text Editor for an example of how to use this. This is the recommended way of multithreading in Blender!
• The multiprocessing module: Allows you to utilize multiple GPU/CPU cores for your rendering. This can have the additional advantage of allowing you to greatly speed up your rendering times by rendering multiple images in parallel.
• The subprocess module: This is by far the simplest way to thread, but only supports command-line operations. If you are willing to do command-line rendering, this should work.
• The concurrent.futures module: This module is a high-level way of operating parallel threads, that essentially allows you to "queue" jobs to be completed. I haven't used this yet, since it's new in Python 3.x, but I recommend you take a look at it to see if it does what you're looking for.

Answer 3

My current solution to this problem, ugly as it is, is to have a modal operator watch output file's size at interval. Size doesn't change = render is finished, which works here because I'm only rendering audio files, so the process should be fast enough. Here's the relevant chunk of the code:

def check_render_file(self, context):

    # ... initialization ...

    if not self.prev_stat:
        self.prev_stat = os.stat(context.scene.render.filepath)

        return {'PASS_THROUGH'}

    cur_stat = os.stat(context.scene.render.filepath)

    if self.prev_stat.st_size != cur_stat.st_size:
        self.prev_stat = cur_stat

        return {'PASS_THROUGH'}

    # ... cleanup, next render ...

    return {'FINISHED'}

def modal(self, context, event):
    if event.type == 'TIMER':
        return self.check_render_file(context)

    return {'PASS_THROUGH'}