Mesh silhouette as a polygon or pixels surrounded by the silhouette on the rendered image?

Question

Here is my target application. The rendered image with rendered meshes is used in an interactive application. A user may click or point the cursor to an object on the image. Information related to that object appears. So I need to know either

1) the coordinates of the polygon of the mesh silhouette on the rendered image

OR

2) pixels on the rendered image surrounded by the silhouette.

I noticed the mesh silhouette is drawn by default in the object mode. But how to find its coordinates from python for the rendered image?

For the 1) my idea was to use bpy_extras.object_utils.world_to_camera_view(..) to find image coordinates of each vertex of the mesh. Then somehow compute the silhouette. For the sake of simplicity, let's neglect transparency and consider that the mesh doesn't have holes. If you can propose a solution for a general case for a mesh with holes (e.g. a torus), that would be great!

Any hint is appreciated

Answer 1

You can use object index to identify to which object does the pixel belongs to, for this you have to :

• assign unique object index to each object
• enable the object index pass in the render layer

after that you can output this read the corresponding object for each point from this pass

---

in the following script we assign unique object index for each object and store this index and name in a dictionary :

import bpy

index = 0
obj_index = dict()

for obj in bpy.context.scene.objects :
    if obj.type == 'MESH' :
        index += 1
        obj.pass_index = index
        obj_index[obj.name] = index

print(obj_index)

---

this part is for extracting the object index pass from the node viewer image :

# division by 4 since we have RGBA
# object index is contained in any of R, G, B values
l = len(bpy.data.images['Viewer Node'].pixels)//4
table = [0 for x in range(l)] 
pixels = bpy.data.images['Viewer Node'].pixels
for i in range(l):
    table[i] = int(pixels[i*4])

print(table[:])

to make this work you have to setup the composition nodes as this :

Answer 2

I guess a custom Freestyle Python script could be a solution:

from freestyle.chainingiterators import *
from freestyle.predicates import *
from freestyle.shaders import *
from freestyle.types import *


class StrokePrinter(StrokeShader):
    def shade(self, stroke):
        for svert in stroke:
            print(svert.point)


upred = AndUP1D(QuantitativeInvisibilityUP1D(0), pyNatureUP1D(Nature.SILHOUETTE))
Operators.select(upred)
Operators.bidirectional_chain(ChainSilhouetteIterator(), NotUP1D(upred))
shaders_list = [
    SamplingShader(5),
    ConstantThicknessShader(4),
    ConstantColorShader(0, 0, 0),
    StrokePrinter()
    ]
Operators.create(TrueUP1D(), shaders_list)

Copy and paste the script into the Text Editor in Blender. Change the Freestyle control mode from the Parameter Editor mode (default) to the Python Scripting mode. Add a style module and select the script from the pull-down menu. Render a still image, then in the console window you will see coordinates of silhouette lines (at every 5 pixels along the silhouette lines; change the parameter of the SamplingShader in the code to tune the interval).

Update: To retrieve the object name of a stroke:

class StrokePrinter(StrokeShader):
    def shade(self, stroke):
        sv1 = stroke[0].first_svertex
        sv2 = stroke[0].second_svertex
        fedge = sv1.get_fedge(sv2)
        print(fedge.viewedge.viewshape.name)
        for svert in stroke:
            print(svert.point)

Answer 3

Here are the algorithms I gathered for the solution 1 (mesh silhouette as a polygon).

• 2d outline algorithm for projected 3D mesh
• Silhouette Edge Detection Algorithms for use with 3D Models
• Introduction to 3D Non-Photorealistic Rendering: Silhouettes and Outlines

Keywords for search: mesh silhouette detection

Mesh silhouette is always displayed in the object mode. It's also calculated by Frestyle. But how to get the silhouette coordinates in the image space?

Answer 4

I might be misunderstanding, but you might be able to do a few simple manipulations to get the outline.

1. Render with transparent film

2. Use this node setup to get the edges of the meshes (the output of the 'greater than' thresholding node are your edges):

3. Add the edges to the render: