Is there a way to do this in Blender on Windows?

If this specific one isn't a capability, are there other neural net things that I would be able to experiment with in/with Blender?

With that example, I'd like to be able to model ideas I had for new manual wheelchairs, and if the program can learn to use them, then maybe I can have it built in real life.


You should be able to use neural networks in Blender for instance with Tensorflow and Keras. There are several problems though, while Blender has several ways to simulate physics, it does not have muscle simulation. The interaction between neural network an physics may be very complicated. Adjusting the state based on the output of the neural network will require different adjustments to depending on the physics sim. For some there may not even be a way to adjust parameters mid-way. Interactive display and performance will require that the NN is run in a different process so the UI isn't locked.

It may be technically possible, but as of now this would require extensive work and likely modification of Blender's source.

  • $\begingroup$ I tried doing some simple optimization with tensorflow. But getting feedback from the scene through a py_function was not possible? Have you actually tested your answer? $\endgroup$ – Leander Aug 6 '19 at 7:33
  • $\begingroup$ @Leander I haven't used Tensorflow directly (always through Keras API) or tested this atm. What feedback should or shouldn't be possible? Everything that's accessible from Blender's Python API can be used to read or modify the state. $\endgroup$ – Robert Gützkow Aug 6 '19 at 7:41
  • $\begingroup$ Afaik they are the same backend. I couldn't read and write data to the scene from a tensorflow graph. I troed setting bone positions through tensors and reading mesh deformations, but it wouldn't work. Ill try again later. If you could provide a minimal working example, that would be great. $\endgroup$ – Leander Aug 6 '19 at 8:08
  • $\begingroup$ @Rjg I just saw this example on youtube of tensorflow, thats amazing if it was made with the combination! youtube.com/watch?v=xmCH-i9Vfw4 $\endgroup$ – Starius Aug 6 '19 at 8:33
  • $\begingroup$ @Leander might not work from within the Tensorflow graph (different process, namespace, interpreter?). It should be possible to use the output of the network, both for calculating the loss and predicting the output after the training is completed. I likely don't have time to do that today, perhaps on the weekend. $\endgroup$ – Robert Gützkow Aug 6 '19 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.