Delete output image file on error

Question

I am currently supervising a render farm which is working using the simple "create empty file when starting a frame" and "don't overwrite existing files" on a shared network file system approach. Unfortunately, one of the computers bugged out a few days ago (without me noticing) with a Cuda error and kept rendering hundreds of black images. The problem is that these black images prevent other computers from taking over where that one computer failed. The other PCs stopped as they thought there's nothing to do anymore.

Is it possible to set Blender to not create a black image if there is an error, but instead delete the placeholder file?

Answer 1

With a lot of trying and testing I came up with the following solution, which is a bit of a hack but it works so far.

I did not find any direct way to find out if there was an error in the rendered image. So I am resorting to analyzing the rendered image after it has been generated.

def PostRender(scene):
    print("Post render: Frame {} of scene {}.".format(scene.frame_current, scene.name))
    if scene != bpy.context.scene:
        # this is not the main scene. skip
        return

    file_invalid = 1
    try:
        print("Retrieving last output file [{0}]".format(last_output_file))
        image_object = bpy.data.images.load(last_output_file)

        num_pixels = len(image_object.pixels)
        width = image_object.size[0]
        height = image_object.size[1]

        # in every 10x10 pixel rectangle, test one pixel. this makes total 1% of the pixels spread all over the image
        for x in range(0, width-1, 10):
            for y in range(0, height-1, 10):
                xoff = int(random.uniform(0, 9))
                yoff = int(random.uniform(0, 9))
                pix_at = 4*(x+xoff + (y+yoff)*width)
                for i2 in range(3):
                    if image_object.pixels[pix_at+i2] != 0.0:
                        file_invalid = 0
                        break                

    except Exception:
        # error accessing data. file is assumed to be invalid
        pass

    if file_invalid == 1:
        print("Image seems to be invalid.")
        # handle the error case here

bpy.app.handlers.render_post.append(PostRender)

By checking only 1% of the pixels, the loop is very fast even for huge images. If you want to be one hundred percent sure, you can add a second loop afterwards for scanning all the pixels if file_invalid is still 1.

This approach has the obvious shortcoming that it will not work if you have black images in your animation.

Possible options what to do to handle the error:

• Leave a log file entry
• Optionally delete the file
• If you suspect a CUDA error (as in my case) it might be a good idea to reboot the computer
• Restart rendering after reboot

I found the last two points especially interesting and implementing that has increased the productivity drastically, since I don't have much downtime anymore. I will not post the implementation here, since it is out of the scope of the question.