1
$\begingroup$

I know this is really specific, but I don't really know which community to approach here.

I am currently trying to speed up my Python addon for Blender using Numba. As you might know, functions sped up with Numba's njit decorator refuse to take any argument that is not a numpy array or a Python primitive.

The solution to this problem is to write so called StructRefs.

If you want to try it out install Numba into your Blender-Python environment and paste the following code into your clipboard:

import sys
import subprocess
import bpy

# Blender's Python executable
pybin = bpy.app.binary_path_python

# Locate users site-packages (writable)
user_site = subprocess.check_output([pybin, "-m", "site", "--user-site"])
user_site = user_site.decode("utf8").rstrip("\n")   # Convert to string and remove line-break
# Add user packages to sys.path (if it exits)
user_site_exists = user_site is not None
if user_site not in sys.path and user_site_exists:
    sys.path.append(user_site)

import numpy as np

from numba import njit
from numba.core import types
from numba.experimental import structref

from numba.tests.support import skip_unless_scipy


# Define a StructRef.
# `structref.register` associates the type with the default data model.
# This will also install getters and setters to the fields of
# the StructRef.
@structref.register
class MyStructType(types.StructRef):
    def preprocess_fields(self, fields):
        # This method is called by the type constructor for additional
        # preprocessing on the fields.
        # Here, we don't want the struct to take Literal types.
        return tuple((name, types.unliteral(typ)) for name, typ in fields)


# Define a Python type that can be use as a proxy to the StructRef
# allocated inside Numba. Users can construct the StructRef via
# the constructor for this type in python code and jit-code.
class MyStruct(structref.StructRefProxy):
    def __new__(cls, name, vector):
        # Overriding the __new__ method is optional, doing so
        # allows Python code to use keyword arguments,
        # or add other customized behavior.
        # The default __new__ takes `*args`.
        # IMPORTANT: Users should not override __init__.
        return structref.StructRefProxy.__new__(cls, name, vector)

    # By default, the proxy type does not reflect the attributes or
    # methods to the Python side. It is up to users to define
    # these. (This may be automated in the future.)

    @property
    def name(self):
        # To access a field, we can define a function that simply
        # return the field in jit-code.
        # The definition of MyStruct_get_name is shown later.
        return MyStruct_get_name(self)

    @property
    def vector(self):
        # The definition of MyStruct_get_vector is shown later.
        return MyStruct_get_vector(self)


@njit
def MyStruct_get_name(self):
    # In jit-code, the StructRef's attribute is exposed via
    # structref.register
    return self.name


@njit
def MyStruct_get_vector(self):
    return self.vector


# This associates the proxy with MyStructType for the given set of
# fields. Notice how we are not contraining the type of each field.
# Field types remain generic.
structref.define_proxy(MyStruct, MyStructType, ["name", "vector"])

print(MyStruct("a",np.array([])))

The code will run just fine once. Try to execute it a second time and Blender will throw a huge list of errors. Does anyone know why this particular combination of tools works so poorly?

EDIT: Here are the errors I am encountering:

Traceback (most recent call last):
  File "/home/my_username/my_folder/work/Blender/Trees/test.blend/Text", line 84, in <module>
  File "/home/my_username/my_folder/work/Blender/Trees/test.blend/Text", line 48, in __new__
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/experimental/structref.py", line 376, in __new__
    return ctor(*args)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/dispatcher.py", line 434, in _compile_for_args
    raise e
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/dispatcher.py", line 367, in _compile_for_args
    return self.compile(tuple(argtypes))
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/dispatcher.py", line 819, in compile
    cres = self._compiler.compile(args, return_type)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/dispatcher.py", line 78, in compile
    status, retval = self._compile_cached(args, return_type)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/dispatcher.py", line 92, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/dispatcher.py", line 110, in _compile_core
    pipeline_class=self.pipeline_class)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler.py", line 627, in compile_extra
    return pipeline.compile_extra(func)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler.py", line 363, in compile_extra
    return self._compile_bytecode()
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler.py", line 425, in _compile_bytecode
    return self._compile_core()
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler.py", line 405, in _compile_core
    raise e
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler.py", line 396, in _compile_core
    pm.run(self.state)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 341, in run
    raise patched_exception
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 332, in run
    self._runPass(idx, pass_inst, state)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 291, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 264, in check
    mangled = func(compiler_state)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/typed_passes.py", line 442, in run_pass
    NativeLowering().run_pass(state)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/typed_passes.py", line 372, in run_pass
    lower.create_cpython_wrapper(flags.release_gil)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/lowering.py", line 244, in create_cpython_wrapper
    release_gil=release_gil)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/cpu.py", line 161, in create_cpython_wrapper
    builder.build()
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/callwrapper.py", line 122, in build
    self.build_wrapper(api, builder, closure, args, kws)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/callwrapper.py", line 176, in build_wrapper
    obj = api.from_native_return(retty, retval, env_manager)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/pythonapi.py", line 1388, in from_native_return
    out = self.from_native_value(typ, val, env_manager)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/pythonapi.py", line 1402, in from_native_value
    return impl(typ, val, c)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/experimental/structref.py", line 167, in box_struct_ref
    ctor_pyfunc = c.pyapi.unserialize(c.pyapi.serialize_object(obj_ctor))
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/pythonapi.py", line 1363, in serialize_object
    struct = self.serialize_uncached(obj)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/pythonapi.py", line 1334, in serialize_uncached
    data = serialize.dumps(obj)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/serialize.py", line 168, in dumps
    p.dump(obj)
  File "/home/my_username/my_folder/blender-2.83.0-linux64/2.83/python/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/serialize.py", line 305, in save
    return super().save(obj)
  File "/home/my_username/my_folder/blender-2.83.0-linux64/2.83/python/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/my_username/my_folder/blender-2.83.0-linux64/2.83/python/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/serialize.py", line 305, in save
    return super().save(obj)
  File "/home/my_username/my_folder/blender-2.83.0-linux64/2.83/python/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/my_username/my_folder/blender-2.83.0-linux64/2.83/python/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/my_username/.local/lib/python3.7/site-packages/numba/core/serialize.py", line 314, in save
    raise _TracedPicklingError(m)
numba.core.serialize._TracedPicklingError: Failed in nopython mode pipeline (step: nopython mode backend)
Failed to pickle because of
  PicklingError: Can't pickle <class '__main__.MyStruct'>: it's not the same object as __main__.MyStruct
tracing... 
 [0]: <class 'method'>: 140247150191712
 [1]: <class 'tuple'>: 140245505894272
 [2]: <class 'type'>: 140247018660896
Error: Python script failed, check the message in the system console

EDIT2: Following Londo's suggestion I changed the last function define_proxy to define_constructor which left me with the following error message:

TypeError: cannot convert native numba.TreeNodeType(('location', int64), ('parent_obj_id', array(float64, 1d, C)), ('weight', int64), ('weight_factor', float64), ('child_indices', int64)) to Python object
Improve this question
$\endgroup$
2
  • $\begingroup$ Maybe pasting a huge list of errors here would be helpful $\endgroup$ – Andrey Sokolov Sep 28 '20 at 12:43
  • $\begingroup$ @AndreySokolov Sorry, I completely forgot about that. There you go! $\endgroup$ – TheBeautifulOrc Sep 29 '20 at 16:27
1
$\begingroup$

it's nothing wrong with your code. it runs just fine when it's set up correctly.

You only have to encapsulate your code within the Blender Environment.

I'd assume: A new Blender instance was generated as it has been launched. As you called your script its structure was bound to the Blender instance. As you called it the 2nd time the Blender instance remained the same but your script generated a new instance of its own.

since the first instance is tied to the current Blender instance and did not receive a destruct call, it's still running. And the comparison of the new instance id with the tied instance id will always fail.

Proof of working Numba scope

as you can see I only made a simple panel-addon in which your code is embedded. your script is tied to the blender instance and does not generate a new instance of its own as long as its active instance is alive or you close the Blender instance.

you may find a better solution to handle your instances. My solution is only for showing the result:-)

Improve this answer
$\endgroup$
5
  • $\begingroup$ What happens when you try reloading your addon? $\endgroup$ – TheBeautifulOrc Sep 30 '20 at 11:56
  • $\begingroup$ after a little tweaking it does the same thing again. when you alter "structref.define_proxy" to "structref.define_constructor" it will stay in the same instance. you will receive conversion warnings which won't break the code as the vanished instance error did. you only have to tweak the type definition, if you wish. but the code keeps running even after reloading the scripts $\endgroup$ – londo Sep 30 '20 at 22:43
  • $\begingroup$ This is really strange. I instantiate my StructRef objects as part of an Operator, so everything should be deleted as soon as the execution is finished. Note that the error does not occur on multiple executions but rather after reloading my scripts. Exchanging the define_proxy function by define_constructor produced just another error (see edit on my question). $\endgroup$ – TheBeautifulOrc Oct 2 '20 at 17:26
  • $\begingroup$ yeah the warning your receive is what I was hinting to, as I mentioned to tweak the types definitions of your struct. I read in the numba doc that its class destructor is being called automatically when each operation has been finished but you haven't have the possibility to call the destructor manually, by now. what if the finish call is never being send to numba because it's running in a blender instance and its state is never set to finish as it runs now? $\endgroup$ – londo Oct 5 '20 at 9:03
  • $\begingroup$ I do not recive a warning but an error. My code does not run with the method you suggested, unless I misunderstood you. What exactly do you mean with "tweaking type definitions"? $\endgroup$ – TheBeautifulOrc Oct 5 '20 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.