# How to place a surface in the projection plane of the viewport

I am currently working on an addon where two-dimensional mesh surfaces should be automatically placed in the projection plane of the viewport window (not that of a camera) (note: perspective projection is selected as the display mode). The position of the mesh surfaces in the three-dimensional space should be determined by the user’s view through the viewport.The meshes do not have to be permanently connected to the viewport, but should be placed only on user input like clicking a button. The meshes should be created as if the viewport is an image viewer displaying an image. At the places where the image is black, mesh areas are to be created (the conversion of an image file into a mesh is not the subject of this question, I am only interested in finding the best way to place the mesh). So the following requirements are made to the addon:

1st: As already mentioned, the mesh surfaces should be in the same plane in three-dimensional space in which the projection plane of the viewport is located.

2: In addition, the mesh surfaces should be in the correct position within the projection plane in relation to the viewport window.

Here is a short GIF that explains the scheme (The black and white image that is briefly shown is for illustration purposes only. The addon does not need to be able to display it).

I have written a script, with which my intention is approximately converted.

import bpy
import numpy as np
import bmesh

#Get the view_matrix and create its inverse.
area = None
for a in bpy.context.workspace.screens[0].areas:
if a.type == 'VIEW_3D':
area = a
break
viewMat = area.spaces[0].region_3d.view_matrix
viewMatInvert = viewMat.inverted()

#Here I have generated the vertices for the mesh surface. They serve only as an example.
#In the final addon they will be generated by a contour approximation
#function of OpenCV. To simplify the script I hard coded this here.
vertices = np.array([[20, 20, 0],
[20,  0, 0],
[ 0,  0, 0],
[ 0, 20, 0],
[12, 12, 0],
[12,  7, 0],
[ 7,  7, 0],
[ 7, 12, 0],
[50, 20, 0],
[50,  0, 0],
[25,  0, 0],
[25, 20, 0],
[32, 12, 0],
[32,  7, 0],
[27,  7, 0],
[27, 12, 0]],
dtype="int32")

#Here I have created a helper frame. It is used to place the mesh surfaces
#at the correct position inside the projection plane, depending on the viewport window.
for a in bpy.context.screen.areas:
if a.type == 'VIEW_3D':
for r in a.regions:
if r.type == 'WINDOW':
width = r.width
height = r.height

helperFrame=np.array([[(width// 2) *-0.25,(height// 2) * 0.25,0],
[(width// 2) * 0.25,(height// 2) * 0.25,0],
[(width// 2) * 0.25,(height// 2) *-0.25,0],
[(width// 2) *-0.25,(height// 2) *-0.25,0],
[((width// 2) *-0.25)+1,((height// 2) * 0.25)-1,0],
[((width// 2) * 0.25)-1,((height// 2) * 0.25)-1,0],
[((width// 2) * 0.25)-1,((height// 2) *-0.25)+1,0],
[((width// 2) *-0.25)+1,((height// 2) *-0.25)+1,0]],
dtype = "int32")

#The helper frame is added to the vertices array.

#I also hardcoded the edges. Again, this is just an
#example and will happen automatically later.
edges = np.array([[ 0, 1],[ 1, 2],[ 2, 3],[ 3, 0],
[ 4, 5],[ 5, 6],[ 6, 7],[ 7, 4],
[ 8, 9],[ 9,10],[10,11],[11, 8],
[12,13],[13,14],[14,15],[15,12],
[16,17],[17,18],[18,19],[19,16],
[20,21],[21,22],[22,23],[23,20]],
dtype = "int32")

#Here a mesh is created from the vertices (including the helper frame)
#and the associated edges.
mesh = bpy.data.meshes.new("Surface")
mesh.from_pydata(finalVertices, edges, [])
mesh.validate()
obj = bpy.data.objects.new("Surface", mesh)
bpy.context.view_layer.objects.active = obj
bpy.ops.object.editmode_toggle(True)
me = obj.data
bm = bmesh.from_edit_mesh(me)

#The mesh is scaled here to fit in the viewport window.
#The factor by which it is scaled I have chosen by trial and error. It is only an approximation.
bmesh.ops.scale(bm,vec=(0.00001,0.00001,0.00001), verts = bm.verts)

#The mesh is filled with faces.
bmesh.ops.triangle_fill(bm, edges = bm.edges)

#This is the point at which the mesh surface is transformed so that it lies
#within the viewpoint (but not the projectionplane/viewport yet). The viewpoint is behind
#(respectively in front of) the projection plane. Here the mesh is not
#visible yet. For this I used the inverted view matrix.
bmesh.ops.transform(bm, matrix = viewMatInvert, verts = bm.verts)

#To move the mesh surface into the visible area it is moved along the surface normals.
#The factor by which the areas were shifted was also selected here by trial and error.
for f in bm.faces:
bmesh.ops.translate(bm, verts=bm.verts, vec= 0.0002*f.normal )

#Here I delete the helper frame. It is no longer needed.
bmesh.ops.delete(bm, geom=bm.verts[-8:])
bmesh.update_edit_mesh(me)
bpy.ops.object.editmode_toggle(True)


I found the following problems: The mesh always has an offset to the actual projection plane. If I try to reduce the offset I always get into trouble with the scaling of the mesh, so that it still fits into the viewport. I can't get the mesh closer than a certain distance to the projection plane.

I would have the following ideas for implementation: 1: Manipulate the viewMatInvert so that bmesh.ops.transform is sufficient to transform the mesh to the right place. In my understanding this would require: take the focal length as value to move the mesh along its faces normally. Take the dimensions of the viewport as value to scale the mesh to fit exactly into the viewport.

2nd: It seems to me that the bpy_extras submodule provides functions that could be helpful in implementing the addon. In addition, the vertices of the mesh at the beginning are only available as two-dimensional vertices. Is it possible to make screen coordinates out of them and then convert them to 3d coordinates? so far I don't understand anything about the module and can't get any further.

However, I am sure that there will be many other possibilities that far exceed my current horizon. I would be very grateful for ideas and tips. P.S.: As far as I understood, I can’t bring the mesh surfaces closer to the projection plane than the Near Clipping Plane. So for the addon I will choose the value for Clip start as small as possible. P.P.S.: I've created a brief overview here of what I mean by which term, in case I've used words differently than they were intended.