6
$\begingroup$

General question: Is there a reliable way to directly access the OpenGL framebuffer of a View3D for copying?

(Please see the end of the post for more detailed questions)

I need a very fast way to copy the pixel data of a SpaceView3D into an OpenGL texture. The blender internal GPUOffscreen.draw_view3d functionality unfortunately takes 10 to 50 ms per call, which is too slow as can be seen in this short example:

import bpy
import bgl
import gpu
import time

WIDTH = 512
HEIGHT =512

offscreen = gpu.types.GPUOffScreen(WIDTH, HEIGHT)

def draw():
    context = bpy.context
    scene = context.scene

    view_matrix = scene.camera.matrix_world.inverted()

    projection_matrix = scene.camera.calc_matrix_camera(
        context.evaluated_depsgraph_get(), x=WIDTH, y=HEIGHT)

    start_time = time.time()
    
    offscreen.draw_view3d(    # this takes 10 to 50 ms (on a Macbook Pro 2016)
        scene,
        context.view_layer,
        context.space_data,
        context.region,
        view_matrix,
        projection_matrix)

    print("draw_view3d call took ", round((time.time() - start_time) * 1000, 2), " ms")


bpy.types.SpaceView3D.draw_handler_add(draw, (), 'WINDOW', 'POST_PIXEL')

What I achieved so far

The following code accesses the currently bound OpenGL draw buffer (Update note: see first comment below) and reads its pixel to a bgl.Buffer object:

import bpy
import bgl

# Draw function which copies data from the 3D View
def draw(self, context):

    if self.modal_redraw == True:

        # get currently bound framebuffer
        bgl.glGetIntegerv(bgl.GL_DRAW_FRAMEBUFFER_BINDING, self.framebuffer)
        #print("Framebuffer: ", framebuffer)

        # get information on current viewport 
        bgl.glGetIntegerv(bgl.GL_VIEWPORT, self.viewport_info)
        #print("Viewport info: ", viewport_info)

        self.width = self.viewport_info[2]
        self.height = self.viewport_info[3]

        # Write copied data to image
        ######################################################
        # resize image obect to fit the current 3D View size
        self.framebuffer_image.scale(self.width, self.height)
        self.pixelBuffer = bgl.Buffer(bgl.GL_FLOAT, self.width * self.height * 4)

        # obtain pixels from the framebuffer
        bgl.glBindBuffer(bgl.GL_FRAMEBUFFER, self.framebuffer[0]) # POST EDIT 08th Aug 2020: This functions fails for some reason, so it doesn't play a role - but why ... ?
        bgl.glReadPixels(0, 0, self.width, self.height, bgl.GL_RGBA, bgl.GL_FLOAT, self.pixelBuffer)

        # write all pixels into the blender image
        self.framebuffer_image.pixels.foreach_set(self.pixelBuffer)

        # reset draw variable:
        # This is here to prevent excessive redrawing
        self.modal_redraw = False


# Modal operator for controlled redrawing of the image object
# NOTE: This code is only for a more conveniant testing of the draw function
#       If you want to stop the test, press 'ESC'

class ModalFramebufferCopy(bpy.types.Operator):
    bl_idname = "view3d.modal_framebuffer_copy"
    bl_label = "Draw 3D View Framebuffer"

    def __init__(self):
        print("Start example code")

        # init variables
        self.width = 32
        self.height = 32
        self.modal_redraw = False
        self.image_name = "framebuffer_copy"
        self.framebuffer = bgl.Buffer(bgl.GL_INT, 1)
        self.viewport_info = bgl.Buffer(bgl.GL_INT, 4)
        self.pixelBuffer = bgl.Buffer(bgl.GL_INT, self.width * self.height * 4)

        # create or update image object to which the framebuffer
        # data will be copied
        if not self.image_name in bpy.data.images:
            self.framebuffer_image = bpy.data.images.new(self.image_name , 32, 32)
        else:
            self.framebuffer_image = bpy.data.images[self.image_name ]


    # 
    def __del__(self):
        print("End example code")


    # modal operator for controlled redraw of the image
    def modal(self, context, event):
        # stop the execution of this example code if 'ESC' is pressed
        if event.type in {'ESC'}:
            bpy.types.SpaceView3D.draw_handler_remove(self._handle_3d, 'WINDOW')
            print("Removing draw handler")
            return {'CANCELLED'}

        else:

            # set draw variable to update:
            # This is here to prevent excessive redrawing
            self.modal_redraw = True

        return {'PASS_THROUGH'}
        #return {'RUNNING_MODAL'}


    def invoke(self, context, event):
        print("Invoking modal operator")

        # Add the region OpenGL drawing callback
        # draw in view space with 'POST_VIEW' and 'PRE_VIEW'
        self._handle_3d = bpy.types.SpaceView3D.draw_handler_add(draw, (self, context), 'WINDOW', 'PRE_VIEW') # this draws the viewport objects alone (without grid)
        #bpy.types.SpaceView3D.draw_handler_add(draw, (self, context), 'WINDOW', 'POST_VIEW') # this draws the grid alone (without objects)

        context.window_manager.modal_handler_add(self)
        return {'RUNNING_MODAL'}


def register():
    bpy.utils.register_class(ModalFramebufferCopy)


def unregister():
    bpy.utils.unregister_class(ModalFramebufferCopy)


if __name__ == "__main__":
    register()

    # Invoke modal operator for the example code
    bpy.ops.view3d.modal_framebuffer_copy('INVOKE_DEFAULT')

This basically works, but neither the 'PRE_VIEW' nor the 'POST_VIEW' flag of the draw_handler_add() give the complete result. The following screenshot shows the copied image (upper left area) and the viewport (lower left area) it was copied from via the code above.

The following screenshot shows the copied image (upper left) and the viewport (lower left) it was copied from via the code above

For clarification

In the end, I don't want to copy the data into a bpy.types.image object. Instead, I want to copy several SpaceView3D frame buffers into one texture and draw it into another framebuffer for applying another shader. The way my code above approaches this is only, because the blender internal approach GPUOffscreen.draw_view3d is too slow.

My open questions

  1. Is there an alternative to GPUOffscreen.draw_view3d that is faster (i.e., < 1 ms)?
  2. Is there a more reliable (maybe blender internal) way to access the 3D View framebuffer?
  3. How can I access the final framebuffer drawn and used by blender (the one where all controls, etc. are implemented)?
  4. The framebuffer I currently access seems to be the object pixel data before any "Color management" settings are applied. Therefore my copied image is much darker than the original 3D View (see attached image). If nobody has an answer to question 3, maybe someone can hint me at how to transform the copied framebuffer data to the correct color space?
$\endgroup$
3
  • $\begingroup$ I now observed something very strange: the bgl.glBindBuffer(bgl.GL_FRAMEBUFFER, FBO) always fails with bgl.glGetError() = GL_INVALID_ENUM. This is true in my example code above and also in any other code I tried. Can someone explain this? Or does anyone have a working example which includes a bgl.glBindBuffer call? I couldn't find one yet ... $\endgroup$ – reg.cs Aug 7 '20 at 22:11
  • $\begingroup$ There is an example Python script, but I haven't tested it. $\endgroup$ – Robert Gützkow Dec 6 '20 at 12:50
  • $\begingroup$ @reg.cs You get GL_INVALID_ENUM because you cant use GL_FRAMEBUFFER with glBindBuffer, to bind framebuffer you probably need to use glBindFramebuffer. $\endgroup$ – scicyb Jan 9 at 15:52
1
$\begingroup$

There is currently no alternative to GPUOffscreen.draw_view3d. For security reasons the internal frame buffer isn't accessible via python. Scripts/Add-ons could change the internal behavior without the user knowing what happens. When using stereo views there are multiple framebuffers to be aware off. Currently if you want access to the framebuffer you need to make changes directly in the draw manager.

The framebuffer of the 3d viewport has no color management applied. Only during drawing to the screen the color management is applied. External render engine can use the bind_display_space_shader to use the color management shader.

$\endgroup$
2
  • $\begingroup$ Adding that, if you do happen to use GPUOffscreen.draw_view3d, the color texture might look "darker" than what you see on the 3D view, because it's in linear RGB space. To make it look like the 3D view you need to process each pixel of that color texture (you can do this with Python, looping over the bgl.Buffer data or an image.pixels list) using the same formula found in the linearrgb_to_srgb() function of the gpu_shader_image_overlays_merge_frag.glsl file, located in the Blender source code (you can find it on Github): /source/blender/gpu/shaders/gpu_shader_image_overlays_merge_frag.glsl $\endgroup$ – R. Navega Feb 15 at 4:33
  • $\begingroup$ For conveniency, here's the Python port of that function. The first version is if you can guarantee that c is never below 0.0, like when reading GL pixels that are clamped to [0, 1]: GAMMA = 1.0 / 2.4 and linearrgb_to_srgb = lambda c: c * 12.92 if c < 0.0031306684425 else 1.055 * c**GAMMA - 0.055 ... The second version is if you need to clamp the input channel value yourself: GAMMA = 1.0 / 2.4 and linearrgb_to_srgb = lambda c: (c * 12.92 if c > 0.0 else 0.0) if c < 0.0031306684425 else 1.055 * c**GAMMA - 0.055 ... You call this function on the R, G and B values of the pixel. $\endgroup$ – R. Navega Feb 15 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.