Resuming rendering image sequence after blender crash

Question

I usually render my animation into image sequence format, It usually takes a lot of time, so I turn off my monitor and come back later to check the rendered result then find that blender crashed before rendering all the frames.

Since rendering can be done from the command prompt I thought about writing a script (running in my OS) to automatically restart rendering from the last successful rendered frame until all frames are finished*.

But I believe this problem is facing a lot of people out there, is there a way to deal with it? instead of reinventing the wheel?

---

• for example if the total render time is 8 hours I don't want come back to check my computer every 20 minutes (some times this is not available I'm asleep or away from my PC)

Answer 1

Blender actually can do this already; if you disable Overwrite in Render settings > Output, Blender will skip rendering frames where the output file already exists.

See Does Blender resume rendering after returning from hibernation in Windows 7?

---

On linux you could write a short shell script to restart the render in case of a crash, e.g.

#!/usr/bin/env bash
while true; do
    blender $@ # $@ contains all the arguments passed to the shell script
done

Save it in a file (e.g. persistent_blender.sh), give it execute permissions and run it like so:

./persistent_blender.sh -b '/path/to/file.blend' -a

Where the arguments tell blender to render the animation of the specified file.

Of course you could make something much nicer and handle all sorts of things (such as stopping when all the output files exist and have a size > 0).

Answer 2

I would write an Autohotkey watchdog script for that. Save this code as watchdog.ahk or smth.

1. Uncheck overwrite images,
2. open Blender file,
3. then run this script, it will show the popup,
4. start initial rendering manually and go to sleep =)

It'll restart Blender if it crashes, open the same file, and send ctrl-f12 again. /repeat/

#SingleInstance
TIMER = 5000
title=
file=
WinWaitActive, ahk_class GHOST_WindowClass
WinGetTitle, title, A
file := RegExReplace(title, ".*\[(.*)\]$", "$1")
if file=
{
    msgbox, CANNOT GET FILENAME FROM TITLE: "%title%"
    exitapp
}
TrayTip, blender, % "Now watching for " file
Loop
{
    IfWinNotExist, %title% ahk_class GHOST_WindowClass
    {
        IfWinExist, ahk_exe blender.exe
            goto next
        TrayTip, blender, % "Restarting " file
        Run, C:\Program Files\Blender Foundation\blender\blender.exe %file%
        WinWaitActive, ahk_class GHOST_WindowClass
        Sleep, 2000
        Send, ^{f12}
    }
next:
    Sleep TIMER
}
+0::exitapp

PS: Pressing Shift-0 closes the script and removes it from Tray.

Answer 3

Here is an example of how this may be achieved using python (version 3.6+).

The script runs Blender in a loop until all files are rendered. If blender crashes the missing file is rerendered. If Blender freezes a timeout restarts the process (I limit render time in Blender by setting Render Properties|Render|Time Limit).

The example is for mac. On windows and linux you need to make sure that path to Blender install folder is stored in the PATH variable.

https://docs.blender.org/manual/en/3.4/advanced/command_line/index.html

import os
import subprocess
from glob import glob

class Data:
    pass

def main():

    data = Data()
    data.start_frame = 1
    data.end_frame = 250
    data.blender_path = "/Applications/Blender.app/Contents/MacOS/Blender"
    # data.blender_path = "blender" # on windows and linux
    data.project_path = "path_to_project/example.blend"
    data.out_path = "path_to_output_folder"
    data.next_frame = 1
    data.render_timeout_seconds = 120

    state = "INIT"

    while True:
        print(state)

        if state == "INIT":
            state = "CHECK_OUTPUT"
            continue

        if state == "CHECK_OUTPUT":
            if are_files_rendered(data):
                state = "DONE"        
            else:
                state = "RUN_BLENDER"
            continue

        if state == "RUN_BLENDER":
            start_blender(data)
            state = "CHECK_OUTPUT"
            continue

        if state == "DONE":
            break

def are_files_rendered(data):

    expected_files = set([f"{frame:04}.png" for frame in range(data.start_frame, data.end_frame + 1)])
    existing_files = set([os.path.basename(path) for path in glob(data.out_path+"[0-9][0-9][0-9][0-9].png")])
    missing_files = sorted(list(expected_files - existing_files))
    
    if missing_files == []:
        return True
    
    def file_name_to_frame_number(file):
        return int(file.split(".")[0])

    data.next_frame = file_name_to_frame_number(missing_files[0])

    return False


def start_blender(data):

    command = f"{data.blender_path} -b {data.project_path} -o {data.out_path}#### -f {data.next_frame}"
    print("\t"+command)

    try:
        subprocess.run(command, shell=True, timeout=data.render_timeout_seconds)
    except subprocess.TimeoutExpired:
        print("TIMEOUT")


main()
```

Answer 4

I encountered the same rendering crashes and resolved them by creating a small Java tool which monitors blender and restarts it externally if the blender process disappears. As the problem seems to be quite widespread, I decided to provide the tool as freeware. You can get it here:

http://zeug.waldpfa.de/blender-tender.php

I also made the observation that - all other things being the same - blender crashes much less frequently when rendering from the command line instead of in the user interface. So if you are rendering with Ctrl-F12, I would recommend to first try switching to command line rendering with

blender -b scene.blend -a

If that does not solve the problem already, you might use Blender Tender to get blender restarted externally and not loose any time or sleep with overnight renders.

As for the question of why it is crashing: As it is much more stable without the user interface, this hints at some rare synchronisation problems between render engine and UI/preview window.