how to get 2d bounding box(x_min,x_max,y_min,y_max values) from IndexOB output of RenderLayer node in Blender compositor?

Question

I'm doing synthetic data generation for AI training. I need to render lots of images a scene, and giving the 2d bounding box value of each object in the image(assuming that all object in the scene will show up in the image). By setting up pass index for each object and enable Object Index form output-passes--data panel. with compositor nodes and a python script i could achieve it.

Here is the code:

import bpy

import numpy as np

import json

import os

def getVisibleBoundingBox(number_of_instances):

# build pass index list
id_list = [n for n in range(1,number_of_instances+1)]

# initiate a dict
gt = {}

S = bpy.context.scene
width  = int( S.render.resolution_x * S.render.resolution_percentage / 100 )
height = int( S.render.resolution_y * S.render.resolution_percentage / 100 )
depth  = 4


#op =bpy.context.scene.node_tree.nodes['Render Layers'].outputs['IndexOB']
pixels = np.array( bpy.data.images[-1].pixels[:] ).reshape( [height, width, depth] )

pixels = np.array( [ [ pixel[0] for pixel in row ] for row in pixels ] )

for id in id_list:
    bbox = np.argwhere( pixels == id)
    #print(bbox)
    if len(bbox) == 0:
        return None
    else:
        (ystart, xstart), (ystop, xstop) = bbox.min(0), bbox.max(0) + 1
        gt[id] = str(xstart), str(xstop), str(height - ystop), str(height - ystart)

#print(gt)
return gt

getVisibleBoundingBox(7)

This uses a viewer node to catch the IndexOb data and turn it an image, and turn the picture into number array to get the 2D-Boundingbox for each object. The thing is, when i run this script(within this scene) in background mode. The viewer node image won't be available. So i'm looking for a way to get the 2D-Boundingbox info directly from IndexOB.

Answer 1

Python access for render ID pass data using files

• The "file output" node needs to be set to export data in raw SRGB without any further adjustments.
• Data is expected to be in the float range of [0..1] and gets mapped to the 8-bit-range [0..255] - every ID above "1" will thus experience clipping. To solve the issue in your comment, divide them by 255 first.
• Make sure that the "Dither" under "Scene"->"Post Processing" is disabled.

After rendering, the ids in the resulting file can be verified with an image editor. (I am working with an animation, so the image itself is just plugged in the "composite" node because it gets saved anyway - depending on how you render, you may have to add a second file output.)

After determining the name and path of our ID pass file, we can load it in to memory. Aside from now having to multiply it by 255 again, the access is similar to using the viewer node and should not affect the rest of your code.

index_file = "ImageIndices" + str(bpy.context.scene.frame_current).zfill(4) + ".png"
index_file_path = "C:\\tmp\\ImageGenIndices\\" + index_file
bpy.data.images.load(index_file_path)
pixels = bpy.data.images[index_file].pixels
idMask = np.multiply(np.array(pixels[:])[0::4].reshape((height,width)),255).astype(int)

(the last line discards the unused color/alpha channels, but aside from the multiplication it is identical to your code)