Fast streaming camera view to video for linux (v4l) system device

Question

What i'm trying to achieve is to stream an arbitrary camera view in blender to a v4l device.

What i've done so far

After reading the community post about streaming the Eevee view on Windows and linux, I adapted the spout addon for blender in order to work with the virtualvideo package. In short, I use a draw handler in Blender to read the pixels the 3D view and send it to the virtual v4l device:

scene = context.scene
dWIDTH = camera.texshare.capture_width
dHEIGHT = camera.texshare.capture_height

view_matrix = context.scene.camera.matrix_world.inverted()

projection_matrix = context.scene.camera.calc_matrix_camera(
    context.evaluated_depsgraph_get(), x=dWIDTH, y=dHEIGHT)

# Defined somewhere else, just here for complete example
offscreen = gpu.types.GPUOffScreen(dWIDTH, dHEIGHT)

offscreen.draw_view3d(
    scene,
    context.view_layer,
    context.space_data,
    context.region,
    view_matrix,
    projection_matrix)

bgl.glDisable(bgl.GL_DEPTH_TEST)
draw_texture_2d(offscreen.color_texture, (0, 0), dWIDTH, dHEIGHT)

# Read pixels and store them in the buffer
bgl.glReadPixels(0, 0, dWIDTH, dHEIGHT, bgl.GL_RGB, bgl.GL_UNSIGNED_BYTE, buffer)

# Send next frame
fvd.send_next_frame(buffer)

What is working

When streaming is enabled, the virtualvideo streams the pixel buffer to the v4l device and i can watch the "stream" with ffplay (or something else):

What isn't working

• The bgl.glReadPixels function takes about 300 to 400 ms to render one frame which results in a slow frame rate.

• By using the view and projection matrices, the output is not exactly the "camera view", even the camera mesh gets rendered into the view

• The rendered image somehow gets displayed in my blender viewport (See image below).

• I somehow managed to flip the image upside down, but that's the least of my problems :D

What i'm asking

• Is there a way to grab and redirect the already created (and very performant) camera view from blender? This does not even necessarily have to be done via the Python interface of Blender, if there is a good solution in C++ or similar I would be very grateful!

• If not, is there an alternative for the bgl.glReadPixels function that works faster?

• Why does the rendered image show up in the viewport and how can i disable this behavior?

Answer 1

I think I made progress which fits to your question.

With a modal operator I have achieved streaming a live camera view from the active Viewport to a v4l device with about 11 fps. Not that smooth, but a good start.

Of course I did not come up with this totally on my own, but i was guided by related threads or answers:

• Question from bluejayke on realtime export of images
• Brilliant Answer from CodeManX on question regarding screenshots

This method was tested in blender 2.82, in version 2.92 it somehow only delivers black images

Workspace "Configuration":

• have a 3D View
• switch to Camera mode (Num 0)
• Zoom Camera to 1:1 (View->Navigation->Zoom Camera 1:1)
• toggle all overlays, gizmos and helper bars which overlap your camera view as they will show in your stream

These steps are not required for the pipeline to work, however what you see in your viewport gets cropped to the render size, so you may have unexpected output if the view doesn't match the camera view 1:1 or has a smaller size than your set resolution

What my code does:

I set up a modal operator which runs at 30Hz (in my use-case it also updates the scene and the camera postion but this is not included in the example).

With the ffmpeg-python package I set up a stream to the previously created virtual video device (setup via v4l2loopback).

In the modal function I grab the pixels in the camera region and put them into the ffmpeg stream.

Example:

import bpy
import bgl
import ffmpeg

class VirtualStream(bpy.types.Operator):
    bl_idname = "wm.virtual_stream"
    bl_label = "Virtual Stream"
    
    _timer = None
    _updating = False
    
    ffmpeg_proc = None
    _region = None
    _res_x = None
    _res_y = None
    
    # change to the path of your v4l device
    _video_device = "/dev/video2"
    
    def modal(self, context, event):
        if event.type in {'ESC'}:
            self.cancel(context)
            return {'CANCELLED'}
        if event.type == "TIMER" and not self._updating:
            self._updating = True
            # correct start pixel of region to camera corner
            x = self._region.x + (self._region.width - self._res_x) // 2
            y = self._region.y + (self._region.height - self._res_y) // 2
            bpy.ops.wm.redraw_timer(type="DRAW_WIN_SWAP", iterations=1)
            # read pixel values in buffer
            buf = bgl.Buffer(bgl.GL_BYTE, self._res_x * self._res_y * 3)
            bgl.glReadPixels(x, y, self._res_x, self._res_y, bgl.GL_RGB, bgl.GL_UNSIGNED_BYTE, buf)
            # convert buffer to bytes for ffmpeg
            buf_list = buf.to_list()
            myBytes = bytes(buf_list)
            # write bytes to virtual video device
            self.ffmpeg_proc.stdin.write(myBytes)
            self._updating = False
        return {'PASS_THROUGH'}
            
    
    def execute(self, context):
        # setup timer for modal 
        wm = context.window_manager
        self._timer = wm.event_timer_add(1/30, window=context.window)
        wm.modal_handler_add(self)
        # get render resolution -> size of camera view
        self._res_x = bpy.context.scene.render.resolution_x
        self._res_y = bpy.context.scene.render.resolution_y
        # setup ffmpeg stream
        self.ffmpeg_proc = (
            ffmpeg.input("pipe:", format="rawvideo", pix_fmt="rgb24", s="{}x{}".format(self._res_x, self._res_y))
            .vflip()
            .output("/dev/video2", format="v4l2", pix_fmt="yuv420p")
            .overwrite_output()
            .run_async(pipe_stdin=True)
        )
        # grab region we want to capture
        for a in context.screen.areas:
            if a.type == "VIEW_3D":
                for r in a.regions:
                    if r.type == "WINDOW":
                        self._region = r
                        break
        self._updating = False
        return {'RUNNING_MODAL'}
    
    
    def cancel(self, context):
        context.window_manager.event_timer_remove(self._timer)
        self.ffmpeg_proc.stdin.close()
        self.ffmpeg_proc.wait()

bpy.utils.register_class(VirtualStream)

Time constraints:

As I mentioned on my machine it runs at about 11 fps, with an additional update of the scene this drops to about 7 fps depending on the updates. When timig the individual parts bgl.glreadPixels took about 50ms and the conversion to bytes also took 50ms, the other parts are virtually irrelevant.

What I don't know is why your execution of bgl.glreadPixel was so much slower than mine or how to speed up from here.