Capture video from cameras in blender game engine for Artificial Intelligence

Question

I'm new to programming in blender game engine. I want to investigate the Q-learning AI using blender game engine. For this purpose I want to make 3d pong. It will look something like this:

The game will contain two boards and one ball. I want to capture data from two cameras and feed it into the Convolutional network and find an optimal actions (Q-values) according them:

I was faced with the problem of obtaining images from cameras in blender game engine. Can anyone help me with this?

Ideally I need transform images to arrays and transfer data using sockets. But, as a last resort, I'll be happy with the option of saving images into folders.

I will not satisfied with the screenshots, using bge.render.makeScreenshot(filename), because I need multiple cameras.

If I succeed in this, I promise to make a tutorial in Youtube about it and share all source files ;)

Sorce files

Many thanks!!!

ADDITION: I've tried to use script from this link:

from bge import logic, render

camList = logic.getCurrentScene().cameras
cont = logic.getCurrentController()
own = cont.owner

cam1 = camList['Camera_LEFT']
cam2 = camList['Camera_RIGHT']

width = render.getWindowWidth()
height = render.getWindowHeight()

cam1.setViewport(0, 0, int(width/2), height)
cam2.setViewport(int(width/2), 0, width, height)

cam1.useViewport = True
cam2.useViewport = True

folder = '//image-#.png'
render.makeScreenshot(folder)

It gives me right view in viewport:

But, screenshot in my directory is still from 3d view (((:

Please, give the source files.

Answer 1

I found the answer!

Thanks ThaTimst3r for his video on YouTube: /watch?v=e4KNstJhFeA&index=1&list=LLNnIYCXmFzdsaND3RsqYSdA

We need to add two planes which we will use for our "screens". These screens need different textures. So, we need to link the materials of screens and cameras. For this purpose, in each screen, we create two properties: the material and the camera.

The values of properties have to be the same as name of material and name of camera respectively.

Next, in our code we will link them:

camera_name = obj['camera_property']

and

matID = texture.materialID(obj, 'MA'+obj['material_property'])

Let's render texure from camera:

renderToTexture = texture.Texture(obj, matID)
renderToTexture.source = texture.ImageRender(scene, cam)

So, final we have the effect what we need. We can even set the resolution of images:

enderToTexture.source.capsize = [500, 500]

And convert images to arrays:

imageArray = texture.imageToArray(renderToTexture.source, 'RGB')

Thats all, I think.

If you have comments, I will listen them with pleasure =)

All source files.