2
$\begingroup$

I can extract a frame from a camera in a scene and process it with numpy but I can't then use the results as a texture. The problem is VideoTexture will only except a path to an image but can't be given that images content directly (ie stringIO).

I tried a few months ago to do this using imagebuff, but with no luck.

As an example of what I'm trying now I've modified the texture swap tutorial from xaedro to apply a crude toon shader using opencv. What I want to do is pass the results of "img4" to "logic.texture.source".

(This requires Opencv2, a camera named "Camera", and an image named "screen" applied to the object the script is attached to. The script is then run as "always"+true pulse.)

from bge import logic
from bge import render
from bge import texture 
import numpy as np
import cv2 


render.setWindowSize(1360, 768) #it will ignore this and default to viewport size    
cont = logic.getCurrentController()
obj = cont.owner
scene = logic.getCurrentScene()
camera = scene.objects['Camera']
new_source = None
render.showMouse(True)
def getTexture(obj):
    ID = texture.materialID(obj, 'IMscreen')
    object_texture = texture.Texture(obj, ID)
    logic.texture = object_texture

def replaceTexture():
    global new_source
    new_source = texture.ImageRender(scene, camera)
    ####################Relevant portion
    img2= np.array(new_source) #export BGE results to numpy
    x,y = new_source.size        
    img3 = np.reshape(new_source, (y,x,4)) #convert to standard image shape


    img4 = cv2.cvtColor(img3, cv2.COLOR_RGBA2BGRA) #Red and blue need swapped for opencv
    img4 = np.flipud(img4) #BGE always gives upside down results. This flips it. 

    #simple Canny toon shader. Just for example, too slow for realtime  
    edges = cv2.Canny(img4,100,200) 
    edges = cv2.cvtColor(edges, cv2.COLOR_GRAY2BGRA)
    img4[:,:,:] = np.where(edges[:,:,:]==255, 0, img4[:,:,:])
    cv2.imshow("result", img4)

    logic.texture.source = new_source #<---want this to be img4 

    logic.texture.refresh(True)       

    ####################End of Relevant portion


if "texture" in obj:
    replaceTexture()

else:
    getTexture(obj)
    obj['texture'] = 1

As a side note there is also the resolution problem (bge.render size is based off viewport not camera size)but I'm ignoring that for now as I think I may have to alter some blender source code.

result

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ imageBuff expects an object that implements the Buffer protocol -blender.org/api/blender_python_api_2_77_0/… $\endgroup$
    – Monster
    Commented Mar 8, 2016 at 10:04
  • $\begingroup$ @Monster that's what I initially tried but couldn't get blender to except what's described in docs.python.org/3/c-api/buffer.html. I'm not very good with C or C structures so I figured I was doing it wrong but I've now tried every possible variation that Blender might want. $\endgroup$
    – Snesticle
    Commented Mar 8, 2016 at 14:16
  • 1
    $\begingroup$ I can provide you Dynamic Texture, which provides some a cleaner approach than what is mentioned at the BGE API docs. The imageBuff implementation is a learning case. It generates a 10x10 color ramp texture. github.com/BlenderMonster/DynamicTexture $\endgroup$
    – Monster
    Commented Mar 9, 2016 at 5:17
  • $\begingroup$ Thank you! Looking at your scripts has gotten me much closer. I got it to accept my buffer now. It's getting an overflow error but I think I know why. I'll post the entire script when I get that fixed. $\endgroup$
    – Snesticle
    Commented Mar 9, 2016 at 12:32
  • $\begingroup$ I've got ValueError: total size of new array must be unchanged , but can't figure out why. $\endgroup$
    – Alan Hao
    Commented Jan 30, 2017 at 22:33

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .