3
$\begingroup$

I have a complex mesh that I am giving custom Normals to via the Data Transfer modifier. The normals are coming from a simpler mesh.

I want to use both the original normals and the custom normals in different parts of the material. Mainly, the original normals for the Diffuse, and the simplified custom normals for the Glossy. However, once I have custom normals of any type, I can no longer access the original normals. The Geometry and Texture Coordinate node Normal outputs both give the Custom Normals. The True Normal gives the original normals, but with flat shading, so that is not helpful.

Is there any method to get at the original normals still? Or a way to modifiy the True Normal in the material to be smooth shaded?

For a static model, this could be achieved by baking either the original or custom normals to a texture and using that. But that is not an option for meshes that deform as far as I can tell?

$\endgroup$
  • $\begingroup$ You can use a script to set the normals to the vertices colors. The downside is, the smooth shading will be Gouraud instead of Phong. $\endgroup$ – Omar Emara Feb 8 '18 at 16:57
  • $\begingroup$ For baking to a texture, you'd want to bake a normal map of the difference between default and custom normals. But you'd have to re-bake it for every pose. Same with vertex colors. $\endgroup$ – Drudge Feb 8 '18 at 20:12
  • $\begingroup$ Animation Nodes can be used to compute the vertex color maps in near realtime, I will write an answer, maybe it will help you get somewhere. $\endgroup$ – Omar Emara Feb 9 '18 at 14:38
  • $\begingroup$ What about duplicating your "complex mesh" inside itself, but assign different vertex group to it. Data Transfer can use vertex group, so you can assign it to yours original mesh, while keep copied mesh with original normals. Just distinguish meshes inside shader with use of vertex colors or texture mask. By multiplying this mask on Normal output, you can extract Normals from copied mesh. And finally use Mix of desired shader and Transparency to hide copied mesh. $\endgroup$ – Serge L Feb 9 '18 at 20:53
7
+50
$\begingroup$

Animation Node can be used to assign vertices normals to vertices colors in near realtime. Using those normals in cycles won't yield the same results as the Phong smooth shading cycles uses, however it provides very similar results.

Background Information

Understanding How Vertex Colors Work

The name Vertex Color is a bit misleading, it should be noted that color data is not stored per vertex, but rather per loop. A loop is basically a vertex of a polygon, so if a vertex is part of four polygons, it will compose four loops, to understand this better, lets look the following illustration:

Loops

The above is a mesh with 4 polygons, each polygon has 4 vertices and subsequently 4 loops. We represented loops by colored dots, you notice that each vertex compose either 4 loops as blue loops, 2 loops as red loops or a single loop as violet loops. Now, If we were to color all loops white expect the blue loop in polygon three, we will get something like this:

Single Loop

There are some advantages of storing colors in loops instead of vertices, the main advantage being the ability to color polygons. If colors were stored in vertices, it would not be possible to color a polygon with a single color, think about it ! To color a polygon using loops, we just have to color all loops in the polygon with the same color:

Polygon Vertex Color

On the other hand, to color a vertex, we have to color all loops that are part of the vertex, same-color dots in the above illustration:

Vertex Vertex Color

In case of coloring vertices, the fragments (pixels or dots) in polygons will be colored according to a convex combination of the colors of its vertices (a bilinear interpolation for quads), such interpolations are particularly useful to represents smooth shading in your case !

So if we were to color vertices based on some color list, we will have to identify the loops that are part of each vertex (same-color dots in our illustration) and color those loops based on corresponding color from the color list. We will also explain polygon coloring for completeness.

Understanding How Loops Are Stored In Meshes

Vertices have indices which we can use to identify and select them by, the same goes for polygons and edges. How are loops ordered and how can we identify them? To answer that, let us consider this simple cube as a study case:

Study Case

Where polygon indices are shown in red and vertices indices are shown in blue. The ordering of loops follows two rules:

  • Loops of the first polygon are stored first, loops of the second polygon are stored next, and so on.
  • Loops in individual polygons are sorted from the lowest vertex index to the highest.

We can represent the loops as an ordered pair $(x, y)$ where $x$ is index of the polygon it is part of and $y$ is the index of the vertices that it is composed of, and the set of ordered pairs for quadrilateral mesh will be:

$$ \begin{aligned} &(0, 0), (0, 1), (0, 2), (0, 3),\\ &(1, 0), (1, 1), (1, 2), (1, 3),\\ &(2, 0), (2, 1), (2, 2), (3, 3),\\ &(3, 0), (3, 1), (3, 2), (3, 3),\\ &\dots \end{aligned} $$

To understand this better, we drew the indices of the loops over the study case :

Loop Indices

Coloring Polygons

In order to color polygons with flat colors, we have to color all loops in a polygon with the same color, in the above case, in order to color the polygon with index zero, we will have to color indices $(0, 1, 2, 3)$, to color the polygon with index one, we will have to color indices $(4, 5, 6, 7)$, and so on. Notice that the indices are in the pattern $(0, 1, 2, \dots)$, grouped in groups of length equal to the number of vertices of the polygon, so all we have to do is generate a list such that the first $n$ number of colors are the color of the first polygon where $n$ is the number of vertices in the first polygons, the second $n$ number of colors are the color of the second polygon where $n$ is the number of vertices in the second polygons and so on. In practice, we loop (iterate) over polygons, get the number of vertices in it ($n$), get its color from the color list, append it to an initially empty list $n$ number of times. An implementation in animation nodes is as follows:

Polygon Implementation

Don't worry about the Set Vertex Color Subprogram, we will study it in a second. Notice that we used the polygon normals as colors in here. Also, we represent colors using 3D vectors; because they in fact are (Vertex Color doesn't support Alpha Channel).

Coloring Vertices

Vertices coloring is a bit harder to understand but easy nonetheless. Lets look at the study case one more time:

Loop Indices

Remember that in order to color vertices, we have to color all loops that compose that vertex, in the above case, $(0, 4, 8)$ should have the color of the vertex at index zero, $(1, 5)$ should have the color of the vertex at index one, and so on. Notice that polygons are merely an indicator of which vertices are in each polygon, now remember that loops are stored in the same order of polygons, so the algorithm for generating the loops colors is as easy as looping (iterating) over polygons, using the indices of the polygon to get the colors at the same indices, append the colors. Take your time to understand it, you will only understand it if you think about it. An implementation in animation nodes is as follows:

Vertices Implementation

Notice that we use the vertices normals as colors in this example.

Set Vertex Color

Animation Nodes currently does not have a node to set vertex colors directly, so we will have to write a small script to do that, a possible way to do it is as follows:

#Create a vertex color map named "Normals" if it is not already created.
if "Normals" in Object.data.vertex_colors:
    map = Object.data.vertex_colors["Normals"]
else:
    map = Object.data.vertex_colors.new("Normals")

#Loop over Colors and assign them to the vertex color map.
for i, color in enumerate(Colors):
    map.data[i].color = color

The script creates a vertex color layer named Normals which is then filled with the data from the Colors list, the Colors list is a 3D Vector list of $n$ Vectors, where $n$ is the number of loops in the mesh.

Answer

You probably already know how to set the vertex normals to vertex color map by now, we did it couple of time above, so such a node tree should achieve what you want:

Node Tree

But wait, there are some things to take care of. Blender only store the colors in the interval $[0, 1]$, but our vectors are signed, ideally in the interval $[-1, 1]$ if normalized. This can be handled by linearly remapping the $[-1, 1]$ interval to the interval $[0, 1]$ in Animation Nodes, then in cycles use the inverse remapping function to transfer the vectors back to the interval $[-1, 1]$. The remapping function can simply be $\frac{V + I}{2}$ where $I$ is the identity vector matrix, and the inverse remapping function will be $2(V - I)$. Notice that $V$ has to be normalized, because while Animation Nodes doesn't care about the magnitude of the normals, cycles does, and will yield unexpected results if not handed normalized vectors. Moreover, normalization has to be done in Animation Nodes due to the assumption we made that the vectors are indeed normalized in the remapping function.

$\endgroup$
  • $\begingroup$ I'm trying to give this a try, but i'm having trouble getting this node setup. Object Mesh does not seem to exist, only Object Mesh Data which doesn't have a Vertex Normals output? I'm using 1.6 as 2.0 doesn't seem to work with blender 2.79 (python version error)? If you can give a bit more info on how this setup works that would also be great. Is Vertex Sampler the above script? If so, what is Vertex Color? $\endgroup$ – Drudge Feb 10 '18 at 3:00
  • $\begingroup$ Looks like 2.0 isn't working because of a custom build I'm using. Do I need 2.0 though? Thanks. $\endgroup$ – Drudge Feb 10 '18 at 3:36
  • $\begingroup$ I found a version of 2.0 that works with my build, and it has the proper node options now. But I still don't really get what's going on, or how to setup the subprograms. $\endgroup$ – Drudge Feb 10 '18 at 9:34
  • $\begingroup$ @Drudge The Object Mesh node does have Vertex Normals, but it is a hidden input, almost all AN nodes have hidden inputs, you can make it visible through the N menu. The Vertex Color is the script node, the Vertex Sampler is the loop. More details will force me to explain things that are way out of the scope of such answer, I will try to provide such details and provide some links to check out in order to understand this better. $\endgroup$ – Omar Emara Feb 10 '18 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.