How to get bounding box of multiple objects in a rendered image with Python?

Question

Hello I am trying to get bounding box of items in my image which is generated automatically using a script by running blender from background. I am using the object index to get the boxes.

This is my current compositor setup,

This is the code I am using to find the boxes

import bpy
import numpy as np

def getVisibleBoundingBox(objectPassIndex,pixels,height,width):
    bbox = np.argwhere( pixels == objectPassIndex )
    (ystart, xstart), (ystop, xstop) = bbox.min(0), bbox.max(0) + 1
    return xstart, xstop, height - ystart, height - ystop 

def getAllBoxes():

    S = bpy.context.scene
    width  = int( S.render.resolution_x * S.render.resolution_percentage / 100 )
    height = int( S.render.resolution_y * S.render.resolution_percentage / 100 )
    depth  = 4

    pixels = np.array( bpy.data.images['Viewer Node'].pixels[:] ).reshape( [height, width, depth] )
    print(max(pixels[540][1]))
    pixels = np.array( [ [ pixel[0] for pixel in row ] for row in pixels ] )
    boxes = []
    for i in  bpy.data.collections['CurrentItems'].objects:
        boxes.append([getVisibleBoundingBox(i.pass_index,pixels,height,width),i.name.split('.')[0]])
    return boxes

print(getAllBoxes())

This all worked fine when tested it with the gui but when running blender as background from command line it just gives this error ValueError: cannot reshape array of size 262144 into shape (1080,1063,4) I believe that is the empty viewer node image thats causing this.

Any ideas on how to get the image or some other method to get the bounding box that would work from background would be most helpful.

I have also tried saving the image as PNG and tried using that but it seems the objectPassIndex data is lost when saving as png.

P.S. I dont want to use world_to_camera_view as there is no reliable way to check for occlusion with that method.

EDIT: The bounding box needs to be a rectangle with the whole object inside it. If a object is occluded by some other object then only the visible part be included in the rectangle.

Answer 1

A solution that worked for me was to use an File Output node instead of viewer node and save the image in openEXR format with full float color depth using this node setup

And then read that image via CV2 library instead of reading it from viewer node image.

Simply replace the following line in the code snippet in question

pixels = np.array( bpy.data.images['Viewer Node'].pixels[:] ).reshape( [height, width, depth] )

with

pixels = cv2.imread(r"filepathOf.exr",  cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)
pixels = cv2.flip(pixels,0)

and it should work fine giving you coordinates of the bounding boxes