rotate objects based on pixel value of video footage

Question

I am currently prototyping an art installation in eevee. Based on a persons silhouette I want knives to rotate 90° (towards the viewer), resulting in a 'mirror', where the unrotated knives make up the bright parts of the wall and the rotated knives reveal a black background.

Here is what I got so far:

How can I animate this concept based on green screen footage?

(just to be clear: the concept doesn't need to be realtime, nor control actual motors. It would serve to try out different knife arrangements and to communicate the idea to donors and technicians)

My first thought was to make a lookup on each frame of the footage, where the individual knife would correspond to a point in UV-space of the video footage. Alpha 0 would leave the knife unrotated, alpha 1 would rotate it 90° on local z. Can I archive this with animation nodes? Or do I need to script? And how could that look like?

Answer 1

Create the alpha mask in the compositor and render it as an image sequence. Then load those images—as separate images—into blender and follow my answer here to access image pixels and orient objects based on it.

To get this to work for the image sequence, adjust the image access expression to be something like this:

bpy.data.images[str(frame).zfill(3)].pixels[:][0::4]

Where frame is your integer frame number. This assumes the images are named 001, 002, 003, ..., adjust the expression to your naming style. This will access a different video frame image based on your scene frame number, effectively giving you the video pixels.