4
$\begingroup$

I have an AI project that I'm working on. I have a machine learning algorithm for a robot, but I want to train it in simulation first. The experimental design is: The robot has a camera, and can see its own hand. It then makes random sequences of motor actions, and creates a causal link between what it saw in its camera, and the random set of motor actions.

I want a setup similar to this: https://www.youtube.com/watch?v=T-c17RKh3uE1

What I want to know is, does the Blender game engine/python library give access to a camera and stream its input in real time, and is it possible to send motor sequences to a virtual arm? Or should I use a different tool? From the virtual camera, I don't require a raytracer, just having access to the pixel buffer is enough using, more over a high resolution isn't required, 1024x768 is plenty.

I can't test my models on a real arm/webcam due to funds. I am good with python, C and C++, I have written many export and mesh scripts for Blender, however I don't know how to set something like this up, or if it is possible. Any help/direction is greatly appreciated.

Thanks

$\endgroup$
  • $\begingroup$ This is possible, but will probably require some very heavy coding in Python. Check out PyroEvil's Neural Network Python scripts for Blender. You can see them in action here and here (more on the channel). $\endgroup$ – Scott Milner Sep 23 '17 at 3:18
  • $\begingroup$ Not sure how up to date this page is. You may want to look at the work being done at open robots $\endgroup$ – sambler Sep 23 '17 at 9:30
  • $\begingroup$ The R&D for my article Torso Crowds was also done with the BGE. It took some changes in the C++ code too, but they were relatively simple additions & fixes, so they were easily accepted and thus are now part of Blender 😀 $\endgroup$ – dr. Sybren Sep 30 '17 at 12:13
4
$\begingroup$

In my mind, blender is a good fit for this sort of project. It has access to python allowing high level development and the use of many many libraries.

What I want to know is, does the Blender game engine/python library give access to a camera

Yes - you can extract the pixels from a camera using the ImageRender refresh() method by passing in a buffer and pixel format. I have used this for a variety of purposes in the past, though not for robotics. https://docs.blender.org/api/blender_python_api_2_78a_release/bge.texture.html#bge.texture.ImageRender.refresh

and stream its input in real time,

You'd have to set something up

and is it possible to send motor sequences to a virtual arm?

Yes. I wrote a minimal test scaffold for a UR5 robot arm. This allowed me to test a program intended for the actual hardware without needing the actual arm present. This was a huge help when trying to do a university project with limited lab time. I also somewhat sidestepped the project (with permission) by writing a blender script to generate the proper commands from the virtual setup....

just having access to the pixel buffer is enough using, more over a high resolution isn't required, 1024x768 is plenty.

That should be possible - although you may find the GPU->CPU bottleneck on some hardware.


Some other things I have done with AI and blender include:

  • Testing IK solutions.
  • GUI's to display telemetry and provide control of various chassis,
  • Prototype control algorithms for hardware (eg four wheel omni-steer chassis - how do you co-ordinate the wheels?)
  • Robot hardware design - doing in in blender+BGE allows it to be tested inside the game engine before building it.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.