# Problem with casting rays from the camera into the scene for each pixel

To retrieve information about the objects that lie behind each pixel, I have to cast rays into the scene for each pixel of a rendered frame.

My problem is that the ray directions which I calculate are not correct.

The eye rays that I generate do not match my image frame vertically (they either start/end outside of it, or too far inside).

For example here is the first direction(top left of the image space) that i calculate And here is the last direction (bottom right of the image space) I think that maybe my calculation of the vertical field of view might be off. But so far I have not been able to fix it.

#calculates the direction the ray has to take for a u, v coordinate
def get_ray_direction(u, v, position, direction, up, right, width, height):
#mapping the u and v coordinates into normalised image coordinates
x = ((2*u - width)/width)
y = ((2*v - height)/height)

direction = x * right + y * up + direction + position
return direction

s = bpy.data.scenes["Scene"]
cam = s.camera
camData = cam.data

view_direction = cam.matrix_world.to_quaternion() * Vector((0.0, 0.0, -1.0))
up = cam.matrix_world.to_quaternion() * Vector((0.0, 1.0, 0.0))
aspect_ratio = s.render.resolution_x / s.render.resolution_y
right = mathutils.Vector.cross(view_direction, up)

up = up.normalized()
right = right.normalized()

#For the FOV I have to use the focal length and sensor size as base values
#since I will be given these values and have to set the camera accordingly
#I took the formula from: http://paulbourke.net/miscellaneous/lens/
fov_v = 2 * math.atan((camData.sensor_height * 0.5) / camData.lens)
fov_h = 2 * math.atan((camData.sensor_width * 0.5) / camData.lens)

up = up * math.tan(fov_v/2)
right = right * math.tan(fov_h/2)

height = s.render.resolution_y/1.0
width = s.render.resolution_x/1.0

#Starting top left, going row by row
for i in reversed(range(s.render.resolution_y)):
for j in range(s.render.resolution_x):
dir = get_ray_direction(j, i, pos, view_direction, up, right, width, height)
ray_dir = dir * 10000
ray_result = s.ray_cast(cam.location, ray_dir);


This is the code that I currently have in total, which contains some debugging stuff like empties that are placed at the ray direction

Here is the primary source I used (among others, but I found this one to be most comprehensible) for the generation of eye rays

## 1 Answer

A solution when you render in 3D View. Maybe something in here will be handy for you. This comes from transforming mouse coords to 3d so maybe its a bit more general than you need.

Getting camera frame corners in screen space (2d):

import bpy
from bpy_extras.view3d_utils import location_3d_to_region_2d

cam = bpy.context.scene.camera
frame = cam.data.view_frame(bpy.context.scene)

# Its in local space so transform to global
frame = [cam.matrix_world * corner for corner in frame]

# Transform into screenspace
region = bpy.context.region
rv3d = bpy.context.region_data
frame_px = [location_3d_to_region_2d(region, rv3d, corner) for corner in frame]


Now you know where exactly the camera frame is and how its panned or zoomed etc. Now you can start firing rays from every pixel in that frame like so:

from bpy_extras.view3d_utils import region_2d_to_origin_3d
from bpy_extras.view3d_utils import region_2d_to_vector_3d

def 2d_to_ray(context, point_px):
region = context.region
rv3d = context.region_data

ray_origin = region_2d_to_origin_3d(region, rv3d, point_px)
ray_vector = region_2d_to_vector_3d(region, rv3d, point_px)

return ray_origin, ray_vector


Now you have a ray vector with origin all in global space for every pixel of camera frame. Here is also how to intersect such ray with a plane (I am using a parallel plane to camera view located at 3d cursor):

def ray_intersect_plane(plane_normal, plane_point, ray_vector, ray_origin):
'''Does not work in Ortho view'''
t = plane_normal.dot(plane_point - ray_origin) / normal.dot(ray_vector)
return line_point + t*ray_vector

# This camera location will work also when there is no camera
# It will work for view-port rendering too
rv3d = context.region_data
cam_loc = rv3d.view_matrix.inverted().translation
cam_look_at = rv3d.view_location
plane_normal = cam_loc - cam_look_at

location = ray_intersect_plane(plane_normal,
bpy.context.scene.cursor_location,
ray_vector,
ray_origin)


If you want to know the pixel dimensions of current area (where the mouse is) for the view-port rendering you do it like this:

regions = bpy.context.area.regions

width = next(region.width for region in regions if region.type=='WINDOW')
height = next(region.height for region in regions if region.type=='WINDOW')


A solution when you render with F12

Don't transform the camera frame points to 2d space. Use the 3d global frame corners and render resolution to interpolate all the render pixels in 3d space.

From camera location and these 3d pixels get your rays. Else is similar.