Is there a way to make node groups with a custom node tree? The option is grayed out in the node menu. I checked Sverchok's implementation, but it also seems like they don't support node groups.
There's a reference implementation for a custom Node Group for CustomNodeTrees, used in Lukas Toenne's object_nodes branch. While part of it is compiled, the UI is implemented using bpy's PyNodes ( A CustomNodeTree, same as Sverchok / AnimationNodes ).
If you are comfortable with Python then having a reference implementation may be enough, the whole concept is a little complex to try to convey as a single answer because it involves describing many sub problems. In short, it is possible, but it's not trivial. you'll have to deal implement a few things:
- How does a node Group behave when copied/duplicated
- Implement the code to generate a node Group from the selected nodes
- Creating a NodeSocketInterface (What's that?!)
Most of these can be copied almost verbatim from the reference implementation but others will likely need some modifications to fit with your own CustomNodetree type.
In Sverchok we recently pushed a new group node implementation (codename Monads): see anything imported by /nodes/basic_data/monad.py. Warning, expect some complex dynamic programming.
It takes a slightly different approach, it is not final but includes a lot of the same functional Operators / UI behaviour:
- socket recast
- delete socket
- rename socket
- make nodegroup from selected.
I don't consider this a reference implementation, but it may be of some interest.