Python API: Multiple types in single UIList, dynamic memory allocation for serialized data [closed]

Question

These questions are about the addon that I'm creating. Please excuse the bad/messy example code, I've been following the "get it working first, then optimize" philosophy.

Issue 1 (dynamic memory allocation):

For my addon, I have defined several custom primitive datatypes (they are all classes derived from bpy.types.PropertyGroup). I want to allow users to add as many of these as they like, so the amount of storage space or data items I will need is unknown (it will be determined at runtime by how many items the user adds).

The only way I know to get blender to manage an unknown number of objects is to create a CollectionProperty (or list) of that object and to call .add() on it, so that new items are added as needed and stored to disk (serialized) by Blender.

Question: Are there better, alternative methods for doing this with the Blender Python API? In C++, I can create a single vector or array of memory addresses/pointers (those pointers pointing to different data types if desired), and then use malloc to request a new empty block of memory to store whatever I want inside, for instance...

Issue 2 (combined list output/pointers):

I want to display multiple types of data in a single unified list in Blender's interface. Every Panel has a UILayout attribute that allows you to define a custom interface layout. If you want to add a list to your panel, you have to use the template_list() function, and you need to supply it with a single CollectionProperty (which is a list of your items).

Because (as stated above) my addon stores user data in multiple lists (one per type of data), I can't directly pass template_list a single list that holds all of my data. To get around this, I have tried to construct a combined list where each item contains a reference to items from the other lists.

In the interface, users add primitive slots with +/- buttons (which adds a pointer item called VEFListItem), then fill the slot with a primitive by clicking on a primitive button directly below the list.

The problem in the code below happens in my AddPoint operator, which attempts to assign a new point primitive to one of the list slots. This doesn't work because VEFListItem has an attribute called data (which is intended to point to an item in one of the primitive lists) that is not a descendent of bpy_struct.

I have also tried making data a bpy.props.PointerProperty(), but in order to do that it seems that I need to specify beforehand what type of data it will be pointing to, and of course I don't know since that is determined at runtime by the user.

Question: Does the Blender API allow me to point to data without first specifying that data's type? Are there alternative approaches here? I know that with this approach, there is the potential problem of stale data with a combined list (which I attempted to get around by reconstructing the list every time a new primitive item is added).

Sample code:

# I will eventually be using these modules
import bpy, bmesh, math, mathutils

# Required addon info (needed to list in the addons panel in user prefs)
# Expand later with remaining entries
bl_info = { "name": "VEF Tools Alpha",
            "description": "Enables arbitrary, precise transformations on mesh elements while modeling",
            "author": "XXXX XXXXXX",
            "version": "0, 1",
            "blender": "2, 68, 0",
            "location": "Properties > Scene > VEF Tools",
            "warning": "Unstable, use at your own risk",
            "category": "Mesh" }
# empty line

# //////////////////////////////
# SECTION: Basic data structures

# Point primitive
class VEFPoint( bpy.types.PropertyGroup ):
    name = bpy.props.StringProperty() # user defined object name

    # coords for this point/vert
    x = bpy.props.FloatProperty()
    y = bpy.props.FloatProperty()
    z = bpy.props.FloatProperty()

    # This class is registered in the register function below


# Line primitive, maybe aggregate VEFPoint types here later? And check precision...
class VEFLine( bpy.types.PropertyGroup ):
    name = bpy.props.StringProperty() # user defined object name

    # Coords of line origin/first point
    x1 = bpy.props.FloatProperty()
    y1 = bpy.props.FloatProperty()
    z1 = bpy.props.FloatProperty()

    # Coords of line destination/second point
    x2 = bpy.props.FloatProperty()
    y2 = bpy.props.FloatProperty()
    z2 = bpy.props.FloatProperty()

    # This class is registered in the register function below


# Plane primitive, maybe aggregate VEFPoint or VEFLine types here later?
class VEFPlane( bpy.types.PropertyGroup ):
    name = bpy.props.StringProperty() # user defined object name

    # Coords of plane origin/first point
    x1 = bpy.props.FloatProperty()
    y1 = bpy.props.FloatProperty()
    z1 = bpy.props.FloatProperty()

    # Coords of plane second point
    x2 = bpy.props.FloatProperty()
    y2 = bpy.props.FloatProperty()
    z2 = bpy.props.FloatProperty()

    # Coords of plane third point
    x3 = bpy.props.FloatProperty()
    y3 = bpy.props.FloatProperty()
    z3 = bpy.props.FloatProperty()

    # This class is registered in the register function below


class VEFListItem( bpy.types.PropertyGroup ):
    data = None

    # # This isn't getting called. Fix.
    # def __init__( self ):
        # self.data = None


class VEFData( bpy.types.PropertyGroup ):
    vefPointList = bpy.props.CollectionProperty( type = VEFPoint )
    vefLineList = bpy.props.CollectionProperty( type = VEFLine )
    vefPlaneList = bpy.props.CollectionProperty( type = VEFPlane )
    vefGuiList = bpy.props.CollectionProperty( type = VEFListItem )
    activeListItemIndex = bpy.props.IntProperty() # stores index of active primitive in my UIList

    # def __init__(self):
        # for item in vefPointList:
            # self.vefGUIList.add( item )
        # for item in vefLineList:
            # self.vefGUIList.add( item )
        # for item in vefPlaneList:
            # self.vefGUIList.add( item )

    def syncList(self):
        for i in range( len( self.vefGuiList ) ):
            self.vefGuiList.remove( 0 )
        for item in self.vefPointList:
            tmp = self.vefGuiList.add()
            tmp.data = item
        for item in self.vefLineList:
            tmp = self.vefGuiList.add()
            tmp.data = item
        for item in self.vefPlaneList:
            tmp = self.vefGuiList.add()
            tmp.data = item
    # This class is registered in the register function below


# END SECTION: Basic data structures
# //////////////////////////////////
# SECTION: Operators


# set the currently active object to a new primitive
class AddPoint( bpy.types.Operator ):
    bl_idname = "vef.addnewpoint"
    bl_label = "Add a new point"
    bl_options = {'REGISTER', 'UNDO'}

    def execute( self, context ):
        print( "NOTHING TO SEE HERE FOLKS." )
        bpy.context.scene.update()
        durp = bpy.context.scene.vef_data.vefPointList.add()
        bpy.context.scene.update()
        print( "DURP: ", durp, "\n" )
        bpy.context.scene.vef_data.vefGuiList[bpy.context.scene.vef_data.activeListItemIndex].data = durp
        bpy.context.scene.update()
        print( "AFTER ASSIGN: ", bpy.context.scene.vef_data.vefGuiList[bpy.context.scene.vef_data.activeListItemIndex].data, "\n" )
        bpy.context.scene.update()

        return {'FINISHED'}


class AddLine( bpy.types.Operator ):
    bl_idname = "vef.addnewline"
    bl_label = "Add a new line"
    bl_options = {'REGISTER', 'UNDO'}

    def execute( self, context ):
        bpy.context.scene.vef_data.vefGuiList[bpy.context.scene.vef_data.activeListItemIndex].data = bpy.context.scene.vef_data.vefLineList.add()
        bpy.context.scene.update()

        return {'FINISHED'}


class AddPlane( bpy.types.Operator ):
    bl_idname = "vef.addnewplane"
    bl_label = "Add a new plane"
    bl_options = {'REGISTER', 'UNDO'}

    def execute( self, context ):
        bpy.context.scene.vef_data.vefGuiList[bpy.context.scene.vef_data.activeListItemIndex].data = bpy.context.scene.vef_data.vefPlaneList.add()
        bpy.context.scene.update()

        return {'FINISHED'}


# Finish later, put in different section
class AddListItem( bpy.types.Operator ):
    bl_idname = "vef.addlistitem" # Check the blender precedent here for naming
    bl_label = "Add a new item"
    bl_options = {'REGISTER', 'UNDO'}

    def execute(self, context):
        item = bpy.context.scene.vef_data.vefGuiList.add()
        bpy.context.scene.update()

        # # Change the active item to the newly added item...will the new item always be last in the collection?
        # bpy.context.scene.vef_data.activeListItemIndex = len( bpy.context.scene.vef_data.vefGuiList ) - 1

        if len( bpy.context.scene.vef_data.vefGuiList ) == 1:
            bpy.context.scene.vef_data.activeListItemIndex = 0

        return {'FINISHED'}


class RemoveListItem( bpy.types.Operator ):
    bl_idname = "vef.removelistitem"
    bl_label = "Remove an item"
    bl_options = {'REGISTER', 'UNDO'}

    #UPDATE THE ACTIVE ITEM INDEX AFTER YOU remove something
    def execute(self, context):
        addonData = bpy.context.scene.vef_data
        if len( addonData.vefGuiList ) == 0 :
            print( "\nNothing to remove\n" )
        elif addonData.vefGuiList[addonData.activeListItemIndex].data == None:
            print( "Muffins are great." )
            addonData.vefGuiList.remove( addonData.activeListItemIndex )
            bpy.context.scene.update()
            if len( addonData.vefGuiList ) < 1 or addonData.activeListItemIndex == 0:
                addonData.activeListItemIndex = 0
            else:
                addonData.activeListItemIndex -= 1
        elif type( addonData.vefGuiList[addonData.activeListItemIndex].data ) == bpy.types.VEFPoint:
            for item in addonData.vefPointList:
                if item == addonData.vefGuiList[addonData.activeListItemIndex].data:
                    addonData.vefGuiList[addonData.activeListItemIndex].data = None
                    addonData.vefPointList.remove()
                    addonData.vefGuiList.remove( addonData.activeListItemIndex )
                    bpy.context.scene.update()
                    if len( addonData.vefGuiList ) < 1 or addonData.activeListItemIndex == 0:
                        addonData.activeListItemIndex = 0
                    else:
                        addonData.activeListItemIndex -= 1

        elif type( addonData.vefGuiList[addonData.activeListItemIndex].data ) == bpy.types.VEFLine:
            for item in addonData.vefLineList:
                if item == addonData.vefGuiList[addonData.activeListItemIndex].data:
                    addonData.vefGuiList[addonData.activeListItemIndex].data = None
                    addonData.vefLineList.remove()
                    addonData.vefGuiList.remove( addonData.activeListItemIndex )
                    bpy.context.scene.update()
                    if len( addonData.vefGuiList ) < 1 or addonData.activeListItemIndex == 0:
                        addonData.activeListItemIndex = 0
                    else:
                        addonData.activeListItemIndex -= 1

        elif type( addonData.vefGuiList[addonData.activeListItemIndex].data ) == bpy.types.VEFPlane:
            for item in addonData.vefPlaneList:
                if item == addonData.vefGuiList[addonData.activeListItemIndex].data:
                    addonData.vefGuiList[addonData.activeListItemIndex].data = None
                    addonData.vefPlaneList.remove()
                    addonData.vefGuiList.remove( addonData.activeListItemIndex )
                    bpy.context.scene.update()
                    if len( addonData.vefGuiList ) < 1 or addonData.activeListItemIndex == 0:
                        addonData.activeListItemIndex = 0
                    else:
                        addonData.activeListItemIndex -= 1
        else:
            print("\nRemove list item failure.\n")

        return {'FINISHED'}


# END SECTION: Operators
# //////////////////////////////////
# SECTION: Addon UI


# Finish this later
class VEFList( bpy.types.UIList ):
    # item = [(),(),(),(),(),]

    # # Enum listing the type of the selected list item
    # VEFType = bpy.props.EnumProperty( items, "Type: ", "Kind of VEF primitive type", default="", options={'ANIMATABLE'}, update=None, get=None, set=None )

    def draw_item(self, context, layout, data, item, icon, active_data, active_propname):
        # Check which type of primitive, separate draw code for each

        if item.data == None:
            layout.label( "Uninitialized object" )
        else:
            layout.label( item.data.name )
# empty line


# Main panel containing all the functionality for the addon
class VEFGui( bpy.types.Panel ):
    bl_idname = "VEF_tools_alpha"
    bl_label = "VEF Tools Alpha"
    bl_space_type = "PROPERTIES"
    bl_region_type = "WINDOW"
    bl_context = "scene"

    def draw(self, context):
        layout = self.layout
        vefDataPtr = bpy.types.AnyType( bpy.context.scene.vef_data )
        addonData = bpy.context.scene.vef_data

        # We start with a row that holds the prim list and buttons for adding/subtracting prims
        dataListRow = layout.row() # Row containing list of prims and +/- buttons
        listCol = dataListRow.column() # Column holding only the list
        listCol.template_list("VEFList", "", vefDataPtr, "vefGuiList", vefDataPtr, "activeListItemIndex", type='DEFAULT' )
        dataManagementCol = dataListRow.column( align = True ) # Column holding the +/- buttons
        dataManagementCol.operator( "vef.addlistitem", icon='ZOOMIN', text="")
        dataManagementCol.operator( "vef.removelistitem", icon='ZOOMOUT', text="")

        # UI elements for the active item in the list
        itemInfoCol = layout.column()
        if len(addonData.vefGuiList) == 0:
            itemInfoCol.label( "Click the + button to add items" )
        elif addonData.vefGuiList[addonData.activeListItemIndex].data == None:
            # Put enum selector here to change the type of this item
            itemInfoCol.label( "Click to make a new item:" )
            newPrims = itemInfoCol.row()

            newPrims.operator( "vef.addnewpoint", icon = 'VERTEXSEL', text="" )
            newPrims.operator( "vef.addnewline", icon = 'EDGESEL', text="" )
            newPrims.operator( "vef.addnewplane", icon = 'FACESEL', text="" )


        elif type( addonData.vefGuiList[addonData.activeListItemIndex].data ) == bpy.types.VEFPoint:
            # Do stuff
            itemInfoRow.label( "This is a point" )
        elif type( addonData.vefGuiList[addonData.activeListItemIndex].data ) == bpy.types.VEFLine:
            # Do stuff
            itemInfoRow.label( "This is a line" )
        elif type( addonData.vefGuiList[addonData.activeListItemIndex].data ) == bpy.types.VEFPlane:
            # Do stuff
            itemInfoRow.label( "This is a plane" )


# END SECTION: Addon UI
# ////////////////////////////////////
# SECTION: Blender required addon code


def register():
    # Add in: check for existing data first. If none found, register stuff to create empties

    # Register classes to make them available for use in Blender
    # primitives + misc.
    bpy.utils.register_class( VEFPoint )
    bpy.utils.register_class( VEFLine )
    bpy.utils.register_class( VEFPlane )
    bpy.utils.register_class( VEFList )
    bpy.utils.register_class( VEFListItem )
    bpy.utils.register_class( VEFData )
    # ops
    bpy.utils.register_class( AddListItem )
    bpy.utils.register_class( RemoveListItem )
    bpy.utils.register_class( AddPoint )
    bpy.utils.register_class( AddLine )
    bpy.utils.register_class( AddPlane )
    # gui
    bpy.utils.register_class( VEFGui )


    # Extend the scene class here, so all scenes now have this data
    bpy.types.Scene.vef_data = bpy.props.PointerProperty( type = VEFData )


def unregister():
    # Check for how to remove bpy.types.Scene.vef_data, put here

    # remove class availability from blender bpy.types
    bpy.utils.unregister_class( VEFData )
    bpy.utils.unregister_class( VEFPlane )
    bpy.utils.unregister_class( VEFLine )
    bpy.utils.unregister_class( VEFPoint )


# END SECTION: Blender required addon code
# ////////////////////////////////////////


register()

Thanks ahead of time, I know this is a complicated post and I appreciate any time spent trying to figure this out.

Answer 1

Issue 1:

CollectionProperty is the only option if you want the properties to be serialized to your .blend by Blender and displayed in a template_list().

You could also use a variable amount of ID properties, but they are limited to primitive types with no support of subtypes, descriptions, nesting etc. You can't really use them for template_list().

You can store everything to user dir or elsewhere with your own code, if you don't need it to be stored in the .blend. Or store to an ID textblock embedded in your .blend, but with the risk of users unlinking or editing it.

Question: What do you mean by primitive / different datatypes? What do your PropertyGroup subclasses represent, and what properties do they have? And what is your addon supposed to do? It looks like you want the user to create mesh geometry within the panel, why?

Issue 2:

You shouldn't display everything in a single list if the data is very different. As you said, there could be a fake list generated from several CollectionProperty lists, which actually store the different kinds of data. Keeping the references (by index?) intact seems challanging, if you want to let the user change order of list items.

Couldn't you use a single PropertyGroup class and store all properties of all your types in here? You would leave certain properties out if the data type doesn't use it, it shouldn't cost much space (if at all). An EnumProperty would indicate what type the current collection item is. It also determines how the template_list is to be drawn.

If you can go without a template_list, it would make things much easier. You would iterate over all CollectionProperty lists and draw UI elements for each item, with almost unlimited freedom of visual appearance (but no scrollable list).

Answer 2

Did not take the time to check the whole code and understand it in depth, but here are two hints:

• We do have a generic RNA pointer type, AnyType (docs), to be used when defining a PointerProperty which might point to unknown/multiple types of data…

• About coordinates, would be better to use a single FloatVectorProperty (docs) instead of three x/y/z FloatProperties, imho.