Introduction
Continuous integration requires a few things, namely tests, a CI server and tooling to run the test suite and read results automatically. Let's take them one by one.
Testing
There are various ways to write unit tests in python, but I decided to go with the classic unittest
module because it is included with blender's python.
While I started off with tests in a different repository, I decided to move them alongside the addon in order to eliminate the need of keeping tests and addon in sync.
Let's say our addon is named toaster
. I would advise naming the tests folder toaster_tests
. You'll see why.
The test suite
The test suite is written in classic unittest
manner.
class ToasterMeshTests(unittest.TestCase):
def setUp(self):
if bpy.data.objects.get('Cube') is not None:
bpy.data.objects.remove(bpy.data.objects['Cube'], do_unlink=True)
def tearDown(self):
pass
def test_addon_enabled(self):
self.assertIsNotNone(toaster.bl_info)
Some caveats:
You need a way to override context because the tests are not always run in the same context as the actual operators. I have found this to mostly work but not always.. Some times you will have to rework your addon in a way that it is less context-sensitive, for example using bmesh
instead of operators to manipulate meshes. You will also need to provide a path for context passing all the way down to the last operator call (which is a good idea anyway as you might find that things break due to context changes when you move your UI around). This context override function seems to work most of the time for me (unlike the one in the API reference)
def context_override(self):
override = bpy.context.copy()
for window in bpy.context.window_manager.windows:
screen = window.screen
for area in screen.areas:
if area.type == 'VIEW_3D':
override['window'] = window
override['screen'] = screen
override['area'] = area
override['region'] = [region for region in area.regions if region.type == 'WINDOW'][0]
override['selected_objects'] = [o for o in bpy.context.scene.objects if o.select]
override['active_object'] = bpy.context.scene.objects.active
return override
- I found a weird bug when reloading the startup file (
bpy.ops.wm.read_homefile()
) in setUp
so for now you need to manually start the suite in an empty file, and clean up after each test.
Graphical version
If your addon is graphical it might make sense to run some of the tests manually and eyeball the result instead of writing complex geometry-understanding scripts. For this reason, I made the tests folder an addon by itself, with a panel and an operator that calls each test separately. This is why I recommend naming the folder toaster-tests
, in order to avoid name conflicts in the blender addons folder.
This is the operator
class TestsOperator(bpy.types.Operator):
bl_idname = "toaster_tests.run"
bl_label = "Toaster Tests"
bl_options = {'REGISTER', 'UNDO'}
testcase = bpy.props.StringProperty()
def execute(self, context):
if self.testcase == "all":
suite = unittest.defaultTestLoader.loadTestsFromTestCase(ToasterMeshTests)
else:
suite = unittest.TestSuite()
suite.addTest(ToasterMeshTests(self.testcase))
unittest.TextTestRunner().run(suite)
return {'FINISHED'}
And the UI panel
#A new panel in the tools section of the 3D view
class View3DPanel():
bl_space_type = 'VIEW_3D'
bl_region_type = 'TOOLS'
class ToasterTestsPanel(View3DPanel, Panel):
bl_label = "Toaster Tests"
bl_context = "objectmode"
bl_category = "Toaster Tests"
def draw(self, context):
layout = self.layout
col = layout.column(align=True)
col.operator("toaster_tests.run").testcase = "all"
col = layout.column(align=True)
col.operator("toaster_tests.run", text="Test 1").testcase = "test_add_column"
col.operator("toaster_tests.run", text="Test 2").testcase = "test_add_beam"
![Toaster tests screenshot](https://i.sstatic.net/0KZU5.jpg)
Running the test from the console
To enable continuous integration you need to be able to run the tests from the terminal. While this is trivial, it is not as trivial to make sure that the CI runner understands if the tests failed or passed. Blender has the --python-exit-code
parameter but this returns non-zero only if an exception is thrown in the supplied python file, immediately before exit.
Thus we need to throw an exception if the tests fail, catch it and immediately exit. This is what this script does.
import unittest
from toaster_tests.suite_1 import ToasterMeshTests
import sys
def run():
suite = unittest.defaultTestLoader.loadTestsFromTestCase(ToasterMeshTests)
success = unittest.TextTestRunner().run(suite).wasSuccessful()
if not success:
raise Exception('Tests Failed')
try:
run()
sys.exit(0)
except Exception:
sys.exit(1)
You can try to run it like this
blender --addons toaster,toaster_tests --python toaster_tests/run.py --python-exit-code 1
You will see blender momentarily launch and output like this:
Read new prefs: /path/to/userpref.blend
Toaster Loaded
Reloaded modules
...
----------------------------------------------------------------------
Ran 3 tests in 0.003s
OK
or if the tests failed
Read new prefs: /path/to/userpref.blend
Toaster Loaded
Reloaded modules
.F.
======================================================================
FAIL: test_add_column (toaster_tests.suite_1.ToasterMeshTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/qwazix/.config/blender/2.78/scripts/addons/toaster_tests/suite_1.py", line 45, in test_add_column
self.assertEquals(bpy.data.objects['Cube'].location, Vector((0,0,-8)), "Column not at the correct position")
AssertionError: Vector((0.0, 0.0, 0.0)) != Vector((0.0, 0.0, -8.0)) : Column not at the correct position
----------------------------------------------------------------------
Ran 3 tests in 0.002s
FAILED (failures=1)
CI!
In the first reference link there's a nice writeup on how to make the CI server (in that case Travis) to download blender in the new VM the CI engine uses. While this allows you to use the shared runners in the case of GitLab (I assume something similar happens with Travis) however I find it wasteful and unnecessarily heavy on blender foundation's servers to download blender each time I push to my server, so I decided to spin up a virtual machine on my local computer.
So go install yourself a distro with your favorite blender version on a virtual machine and then install gitlab-ci-multi-runner
.
If you are wondering why a VM, it's because you might have done all sorts of weird things on your local blender installation (enabled multiple addons, installed python modules and other things) that a user might not have and thus the fact that the tests pass on your machine doesn't mean it will work on a clean environment.
Register your runner with gitlab
gitlab-runner register
It will ask you to provide a token from your gitlab repository and give a name to the runner. For the first time it might be useful to do it as user rather than as root, in order to run it manually and be able to observe what is going on. Afterwards, registering as root allows for unattended tests by just having the VM booted up, though you might run into issues with the X11 display. Choose a name and labels as you like and set executor as shell
.
Now run it (except if you set up as root, where it automatically runs in the background)
gitlab-runner run
CI configuration
To trigger a CI build in GitLab you need to add a .gitlab-ci.yml
in the root of your repository. The following file works for me
job1:
script:
- mkdir -p ~/.config/blender/2.78/scripts/addons/
- rm -f ~/.config/blender/2.78/scripts/addons/toaster
- rm -f ~/.config/blender/2.78/scripts/addons/toaster_tests
- ln -s `pwd`/toaster/ ~/.config/blender/2.78/scripts/addons
- ln -s `pwd`/toaster_tests/ ~/.config/blender/2.78/scripts/addons
- blender --addons toaster,toaster_tests --python toaster_tests/run.py --python-exit-code 1
If you choose to run it as root you will need to append the following switches to the last line, as the runner runs under the gitlab-runner
user which does not have a DISPLAY
, nor access to alsa
.
-b -noaudio
Note that I suspect that running with -b
might incur additional context issues with blender, since it's possible that context override won't work (no windows to loop through in context_override()
). This theory is not tested.
Passed!
Now if all went well, your runner should show
Checking for builds... received build=6237996 repo_url=https://gitlab.com/d-e/blender-addon-boilerplate.git runner=78a379b6
Build succeeded
and the build page in gitlab should show the output from blender. After a while a nice green reassuring
![Passed](https://i.sstatic.net/swKXK.png)
badge should be besides your last commit.
Wrapped and ready to go
I've updated my boilerplate blender addon (now named Toaster) with CI configuration. There are two branches, one that launches the full blender gui in the runner and the nogui-testing
branch which runs blender in the background. Go get it.
References
- Automatic Blender Addon Testing with TravisCI
- Terminate Blender with Exit Code “1” when run from command line
- Blender Manual – Command Line Arguments
- Gitlab runner documentation