Body part recognition inside of blender using a neural net

Question

So I'm working on a project that's turning out to be monstrously difficult. I'm trying to automate the rigging process inside of blender using the addon "Auto-rig pro". This addon has a feature called smart rig that automatically places a skeleton into a mesh. Using this I've been able to automate everything but one step.

In order for the rig to be created automatically you have to manually place markers on the neck, chin, shoulders, wrists, ankles, and groin. If you've ever used mixamo's autorigger it's basically just that. So in order to fully automate the rigging process I've resorted to using a neural net.

I've been doing a lot of research on machine learning, but unfortunately it appears that I'm the first to attempt something like this (as far as I know). After significant effort I've managed to install Tensorflow and Keras into blender, but I start running into problems when I try to figure out how to feed the data to the neural net.

Normally a neural net is fed raw images, but I need to feed it the 3d view. The marker placement is done in a straight on orthographic view, so I only have to worry about 2 dimensions here which is nice. I could render each model as an image and input it into the neural network, then have the neural net output the appropriate marker coordinates, but rendering takes time and this method seems somewhat unreliable.

Honestly I'm feeling pretty damn overwhelmed by this whole process, I'd never written anything in python until a few days ago, and I've never tried to make a neural net do anything more sophisticated than recognizing simple number patterns.

If you have any suggestions or ideas at all I'd love to hear them and I'm happy to answer any questions you have. Just feeling like anyone other than myself is considering this problem would mean a lot to me.

Also I apologize for any typos/grammatical errors I may have made. After working on this whole neural net dilemma for 8 hours straight my brain is pretty much completely fried.

Answer 1

Consider this to your approach:

• feeding your net just a 2d image is limiting (and more work to set it up) when you could feed it whole 3d mesh. You could even build an octree and know what is where in space and feed this to the neural net, basically voxel data instead of pixels

• probably the whole neural net thing is not even necessary. There is a popular method of extracting skeletons from meshes by Laplacian smoothing the mesh: Skeleton Extraction by Mesh Contraction:

  Our method extracts a 1D skeletal shape by performing geometry contraction using constrained Laplacian smoothing.

  You will find this in open source C++ CGAL library here. This is probably enough to know where the neck and crotch is and where limbs end.

• other method is: Mean Curvature Skeletons:

  

  Code here.

Answer 2

You probably want to Access Render Result pixels from a script.

From that question, there are two possible methods, which both result in a numpy image which you can feed to tensorflow. You will probably have to reshape it since tensorflow excepts a non standard shape.

Create an orthographic camera directed at your model.

This first method utilizes the Viewer Node Hack. With it, we can't use opengl, but in my test Blender Render could be setup to run very quickly. I display the image at the end with opencv2, but that wouldn't be necessary.

1. Create a Renderlayer and connect it to the viewer node.
2. Render the image.
3. Read the pixels from the image "Viewer Node"
4. Convert the image to an 8bit bitmap in the $[0, 255]$ range. (Note, that this image will not have any color transformation and will correspond to the RAW setting of Blenders color management.)

The second method works with opengl, but requires saving the image and opening it. We can't use the Viewer Node Hack, because the OpenGL render does not get Postprocessed with Compositing.

1. Render the image with opengl.
2. Save the "Render Result" image
3. Open the image with a python module like Pillow or opencv.

Both methods use rendering, but it can probably be setup to be quick enough. My non-optimized renders took less than 0.4s.

---

1st method

import bpy
import numpy as np

bpy.context.scene.use_nodes = True
tree = bpy.context.scene.node_tree
links = tree.links

for n in tree.nodes:
    tree.nodes.remove(n)

rl = tree.nodes.new('CompositorNodeRLayers')
v = tree.nodes.new('CompositorNodeViewer')
v.use_alpha = False

links.new(rl.outputs[0], v.inputs[0])  # link Image output to Viewer input

render = bpy.context.scene.render
render.resolution_percentage = 100
bpy.ops.render.render()

# get viewer pixels
viewer_image = bpy.data.images['Viewer Node']
pixels = viewer_image.pixels

arr = np.array(pixels[:])
arr = arr.reshape(render.resolution_y, render.resolution_x, 4)
arr = np.flipud(arr)
# at this point the image is loaded into arr in RGBA as 64bit floats

arr*=255                    #convert to 8bit [0, 255]
arr = arr.astype(np.uint8)

# this is just for confirmation, remove if unnecessary
import cv2
bgr = cv2.cvtColor(arr, cv2.COLOR_BGRA2RGBA)
cv2.imshow("img", bgr) # display the image
cv2.waitKey(1)

---

2nd method

import bpy
import numpy as np
import cv2

bpy.ops.render.opengl()
bpy.data.images["Render Result"].save_render("D:/opengl_render.png")
arr = cv2.imread("D:/opengl_render.png")
cv2.imshow("arr", arr)
cv2.waitKey(1)