How to add a Modifier
(This answer is for 2.7X versions, 2.8 has made some changes to modifier code, and I haven't got around to looking at them)
A while back, Campbell Barton created a patch template for a deform modifier, which provides a great place to start when looking at creating a modifier, it can be found here. In general, once you know which files to look in, you can just follow what has been done for other similar modifiers, as it tends to be fairly systematic.
I will assume a working knowledge of Blender's DNA/RNA system (See https://wiki.blender.org/wiki/Reference/FAQ for a starting point if you are unfamiliar)
In essence you need to do the following:
Define a struct and appropriate exports in source/blender/makesdna/DNA_modifier_types.h for your modifier properties. Make sure you follow DNA byte alignment rules (the compiler will check and and help you)
Create your modifier file in source/blender/modifiers/intern/ (And don't forget to add it to CMake). This is the file where you will do most of the work. The most important thing you need to do is to define a ModifierTypeInfo struct which describes how your modifier behaves and provides access to the methods that actually execute it. See other modifiers for examples. This includes indications on what sort of data the modifier accepts, how to construct the modifier data, and depsgraph relationships and many more bits and pieces. Be aware the deform only modifers are different that onces that change vertices.
In source/blender/modifiers/MOD_modifiertypes.h add a reference to the above ModifierTypeInfo
In source/blender/makesrna/RNA_access.h define the RNA mapping for your modifier data DNA struct so that it can be accessed by the UI.
Add a UI for the modifier panel in release/scripts/startup/bl_ui/properties_data_modifier.py
If you have added any pointers in your DNA struct, you will need to update the source/blender/blenloader/intern/writefile.c and source/blender/blenloader/intern/readfile.c files to define save and load behavior. This is also used in undo.
Suggestions for your particular Modifier
For the particular modifier you want to add, you should actually be able to do this by combining two existing modifiers.
- The Vertex Weight Edit modifier supports a texture mask option that you could use to define which vertices you want mask. If it doesn't provide enough control, I would suggest looking at enhancing this modifier.
- Them just use the ordinary mask modifier with the generated vertex group.