# Mesh silhouette as a polygon or pixels surrounded by the silhouette on the rendered image?

Here is my target application. The rendered image with rendered meshes is used in an interactive application. A user may click or point the cursor to an object on the image. Information related to that object appears. So I need to know either

1) the coordinates of the polygon of the mesh silhouette on the rendered image

OR

2) pixels on the rendered image surrounded by the silhouette.

I noticed the mesh silhouette is drawn by default in the object mode. But how to find its coordinates from python for the rendered image?

For the 1) my idea was to use bpy_extras.object_utils.world_to_camera_view(..) to find image coordinates of each vertex of the mesh. Then somehow compute the silhouette. For the sake of simplicity, let's neglect transparency and consider that the mesh doesn't have holes. If you can propose a solution for a general case for a mesh with holes (e.g. a torus), that would be great!

Any hint is appreciated

• – Chebhou May 7 '15 at 0:22
• What is the purpose of this ? ( there maybe some easier ways to do it ) – Chebhou May 7 '15 at 0:24
• Here is my target application. The rendered image is used in an interactive application. A user may click or point the cursor to an object on the image. Information related to that object appears. So I need to know the coordinates of the polygon of the mesh outline on the rendered image. – vvoovv May 7 '15 at 8:59
• if you can just save this pass as one image and read the pixel value which would be the object index directly it would be easier – Chebhou May 7 '15 at 13:00
• it does not have to be bmp i just thought this format is easier , what do think ? you can create 2D array same size as the image and store obj ind in it for fast look-up ( for each frame ) – Chebhou May 7 '15 at 13:09

You can use object index to identify to which object does the pixel belongs to, for this you have to :

• assign unique object index to each object
• enable the object index pass in the render layer

after that you can output this read the corresponding object for each point from this pass

in the following script we assign unique object index for each object and store this index and name in a dictionary :

import bpy

index = 0
obj_index = dict()

for obj in bpy.context.scene.objects :
if obj.type == 'MESH' :
index += 1
obj.pass_index = index
obj_index[obj.name] = index

print(obj_index)


this part is for extracting the object index pass from the node viewer image :

# division by 4 since we have RGBA
# object index is contained in any of R, G, B values
l = len(bpy.data.images['Viewer Node'].pixels)//4
table = [0 for x in range(l)]
pixels = bpy.data.images['Viewer Node'].pixels
for i in range(l):
table[i] = int(pixels[i*4])

print(table[:])


to make this work you have to setup the composition nodes as this :

• How can I access the object index from the related image (IndexOB pass) via python? – vvoovv May 7 '15 at 13:37
• – Chebhou May 7 '15 at 13:40
• Why did you use the "Normalize" node? Working with raw values without any normalization is more convinient. – vvoovv May 7 '15 at 19:49
• i think the image will clip values more than one – Chebhou May 7 '15 at 22:53
• I assigned index 2000 (2 thousand) to an object. It was preserved in any of RGB values. – vvoovv May 8 '15 at 0:06

I guess a custom Freestyle Python script could be a solution:

from freestyle.chainingiterators import *
from freestyle.predicates import *
from freestyle.types import *

for svert in stroke:
print(svert.point)

upred = AndUP1D(QuantitativeInvisibilityUP1D(0), pyNatureUP1D(Nature.SILHOUETTE))
Operators.select(upred)
Operators.bidirectional_chain(ChainSilhouetteIterator(), NotUP1D(upred))
StrokePrinter()
]


Copy and paste the script into the Text Editor in Blender. Change the Freestyle control mode from the Parameter Editor mode (default) to the Python Scripting mode. Add a style module and select the script from the pull-down menu. Render a still image, then in the console window you will see coordinates of silhouette lines (at every 5 pixels along the silhouette lines; change the parameter of the SamplingShader in the code to tune the interval).

Update: To retrieve the object name of a stroke:

class StrokePrinter(StrokeShader):
sv1 = stroke[0].first_svertex
sv2 = stroke[0].second_svertex
fedge = sv1.get_fedge(sv2)
print(fedge.viewedge.viewshape.name)
for svert in stroke:
print(svert.point)

• Thank you for your script! Let's consider a complex example Freestyle example The torus on the image is partially hidden by the cube. How can one find the silhouettes of the cube and the torus? In the latter case the silhouette consists of two polygons. – vvoovv May 19 '15 at 13:26
• In that case you have 3 contour lines: One for the cube, one for the torus silhouette on the left, and another for the torus silhouette on the right. Using the Freestyle Python API, you can retrieve the name of the object to which each contour line belongs to. <pre><code> class StrokePrinter(StrokeShader): def shade(self, stroke): sv1 = stroke[0].first_svertex sv2 = stroke[0].second_svertex fedge = sv1.get_fedge(sv2) print(fedge.viewedge.viewshape.name) for svert in stroke: print(svert.point) </code></pre> – T.K. May 19 '15 at 17:16
• Hmm mini-Markdown does not seem to allow me to insert a formatted code block... Updated the answer to show the revised code. – T.K. May 19 '15 at 17:27
• Then I need to compose polygons out of stroke coordinates. Am I correct that there is no guarantee that a silhouette for each object is a closed polygon (without breaks)? – vvoovv May 19 '15 at 18:38
• No there is no guarantee that object silhouettes are a closed polygon. Check out the Freestyle SVG exporter add-on. It comes with an option to generate fills (closed polygons) out of selected feature lines. You could isolate silhouette lines (possibly together with object names) as illustrated above, and combine them with the SVG fills afterwards. wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.73/Freestyle – T.K. May 20 '15 at 4:13

Here are the algorithms I gathered for the solution 1 (mesh silhouette as a polygon).

Keywords for search: mesh silhouette detection

Mesh silhouette is always displayed in the object mode. It's also calculated by Frestyle. But how to get the silhouette coordinates in the image space?

• is this an edit to the question ? – Chebhou May 7 '15 at 22:57
• My task can have 2 solutions: 1) mesh silhouette as a polygon, 2) pixels surrounded by the silhouette. You helped me to elaborate solution 2). Still, I'd like to have something for solution 1) as well. – vvoovv May 8 '15 at 0:10
• Which edits would you suggest to make? I can't write the answer since I didn't study the algorithms in the links thoroughly. – vvoovv May 8 '15 at 13:05

I might be misunderstanding, but you might be able to do a few simple manipulations to get the outline.

1. Render with transparent film

1. Use this node setup to get the edges of the meshes (the output of the 'greater than' thresholding node are your edges):

1. Add the edges to the render:

• I need a silhouette in the vector form (polygons) for each object in the scene. Your solution provides bitmap silhouette for the scene as a whole. But I tried your setup and learned something new. Thank you anyway! – vvoovv May 19 '15 at 18:29
• @vvoovv ahhh I knew I was missing something. FYI you might try checking out skimage.measure.find_contours and running it over your object index pass. – ajwood May 19 '15 at 18:53