ray_cast function not able to select all the vertices in camera view

Question

The aim of the below code using ray_cast function (thanks to @lemon) is to find all the visible vertices present in the camera view (previous post's Link). It does find some of the vertices of a mesh in the camera view, however, some of the vertices are missed as shown in Figure 1 and 2. But, for my work, I need all the vertices as shown in figure 3 and 4.

import bpy
from mathutils import Vector
from mathutils.bvhtree import BVHTree
from bpy_extras.object_utils import world_to_camera_view

# Create a BVH tree and return bvh and vertices in world coordinates 
def BVHTreeAndVerticesInWorldFromObj( obj ):
    mWorld = obj.matrix_world
    vertsInWorld = [mWorld * v.co for v in obj.data.vertices]

    bvh = BVHTree.FromPolygons( vertsInWorld, [p.vertices for p in obj.data.polygons] )

    return bvh, vertsInWorld

# Deselect mesh polygons and vertices
def DeselectEdgesAndPolygons( obj ):
    for p in obj.data.polygons:
        p.select = False
    for e in obj.data.edges:
        e.select = False

# Get context elements: scene, camera and mesh
scene = bpy.context.scene
cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']

# Threshold to test if ray cast corresponds to the original vertex
limit = 0.0001

# Deselect mesh elements
DeselectEdgesAndPolygons( obj )

# In world coordinates, get a bvh tree and vertices
bvh, vertices = BVHTreeAndVerticesInWorldFromObj( obj )

print( '-------------------' )

for i, v in enumerate( vertices ):
    # Get the 2D projection of the vertex
    co2D = world_to_camera_view( scene, cam, v )

    # By default, deselect it
    obj.data.vertices[i].select = False

    # If inside the camera view
    if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0: 
        # Try a ray cast, in order to test the vertex visibility from the camera
        location, normal, index, distance = bvh.ray_cast( cam.location, (v - cam.location).normalized() )
        # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
        if location and (v - location).length < limit:
            obj.data.vertices[i].select = True

del bvh

I've modified the above code a bit just to test the behaviour of normal scene.ray_cast function (not the bvh.ray_cast). However, the results are same.

import bpy
from mathutils import Vector
from bpy_extras.object_utils import world_to_camera_view

# Deselect mesh polygons and vertices
def DeselectEdgesAndPolygons( obj ):
    for p in obj.data.polygons:
        p.select = False
    for e in obj.data.edges:
        e.select = False

# Get context elements: scene, camera and mesh
scene = bpy.context.scene
cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']

# Threshold to test if ray cast corresponds to the original vertex
limit = 0.0001

# Deselect mesh elements
DeselectEdgesAndPolygons( obj )

# In world coordinates, get vertices
mWorld = obj.matrix_world
vertices = [mWorld * v.co for v in obj.data.vertices]

print( '-------------------' )

for i, v in enumerate( vertices ):
    # Get the 2D projection of the vertex
    co2D = world_to_camera_view( scene, cam, v )

    # By default, deselect it
    obj.data.vertices[i].select = False

    # If inside the camera view
    if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0: 
        # Try a ray cast, in order to test the vertex visibility from the camera
        ray= obj.ray_cast( cam.location, (v - cam.location).normalized() )
        # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
        if ray[0] and (v - ray[1]).length < limit:
            obj.data.vertices[i].select = True

Hence, I'm wondering whether there is some error with the ray_cast functions or something I'm doing wrong.

Answer 1

Has the same problems with raycasts... Seems they are not so reliable on contours. After some digging i just started to shoot several raycasts to small cube around vertex, if any of rays hit the face with this vertex -> vertex is visible

Answer 2

Thank you for the excellent suggestions on this problem.

I was also having similar accuracy trouble w/ BVH hit testing, using Animation Nodes.

Based on your suggestions, I discovered much better results by raycasting the path from the vertex back to the camera, then testing whether the ray hit anything in the BVH. If nothing was hit, it's a clear path to the camera.

HTH

Demo file notes: I'm using an empty called Viewer as a proxy for the camera.

Also, Epsilon set to 0 for speed, but on raycast I've got min distance set to 0.01 units so it doesn't get a false positive hit just adjacent to the origin vertex.

Answer 3

The modified code!:

import bpy
from mathutils import Vector
from bpy_extras.object_utils import world_to_camera_view

# Deselect mesh polygons and vertices
def DeselectEdgesAndPolygons( obj ):
    for p in obj.data.polygons:
        p.select = False
    for e in obj.data.edges:
        e.select = False

# Get context elements: scene, camera and mesh
scene = bpy.context.scene
cam = bpy.data.objects['Camera']
obj = bpy.data.objects['Cube']

# Threshold to test if ray cast corresponds to the original vertex
limit = 0.1

# Deselect mesh elements
DeselectEdgesAndPolygons( obj )

# In world coordinates, get a bvh tree and vertices
mWorld = obj.matrix_world
vertices = [mWorld @ v.co for v in obj.data.vertices]

print( '-------------------' )

for i, v in enumerate( vertices ):
    # Get the 2D projection of the vertex
    co2D = world_to_camera_view( scene, cam, v )

    bpy.ops.mesh.primitive_cube_add(location=(v))
    bpy.ops.transform.resize(value=(0.01, 0.01, 0.01))

    # By default, deselect it
    obj.data.vertices[i].select = False

    # If inside the camera view
    if 0.0 <= co2D.x <= 1.0 and 0.0 <= co2D.y <= 1.0 and co2D.z >0: 
        # Try a ray cast, in order to test the vertex visibility from the camera
        location= scene.ray_cast(bpy.context.window.view_layer, cam.location, (v - cam.location).normalized() )
        # If the ray hits something and if this hit is close to the vertex, we assume this is the vertex
        if location[0] and (v - location[1]).length < limit:
            obj.data.vertices[i].select = True