2
$\begingroup$

I have written a very straight-forward script that imports many obj files one-by-one and renders them. The imported meshes have ~10k to ~120k vertices. After rendering, I completely remove the imported mesh (along with its data blocks) before importing the next one. However, as the for loop proceeds the importing procedure becomes extremely slow. I noticed that the import function starts behaving in weird ways and it takes a lot of time to import objects. I'm not sure why this is happening. Initially I was thinking memory issues are causing this but I think removing the data blocks should resolve memory leakage. This happens even without doing any rendering or any operations on the imported objects. Here's an instance of what the import function prints while importing a mesh:

(  0.0002 sec |   0.0002 sec) Importing OBJ '/data/e1f2651d55aecd7d8f2b6fca0ec9a39dn7a9/models/model.obj'...
(  0.0308 sec |   0.0306 sec) Parsing OBJ file...
(  1.8534 sec |   1.8511 sec) Done, loading materials and images...
(  2.0450 sec |   2.0426 sec) Done, building geometries (verts:72707 faces:137005 materials: 44 smoothgroups:0) ...
(  5.4944 sec |   5.4921 sec) Done.
(  5.4946 sec |   5.4945 sec) Finished importing: 'data/e1f2651d55aecd7d8f2b6fca0ec9a39dn7a9/models/model.obj'
Progress: 100.00%

As more objects are imported the import function behaves slower and slower, even for simpler shapes (e.g. ~12k vertices) you will get something like this:

(  0.0002 sec |   0.0002 sec) Importing OBJ '/data/jjgd2e3f46f7cc1e8ba69a5e14689f7b974/models/model.obj'...
  (  0.0266 sec |   0.0263 sec) Parsing OBJ file...
    (  0.7060 sec |   0.6793 sec) Done, loading materials and images...
    (  3.0993 sec |   3.0726 sec) Done, building geometries (verts:12668 faces:43914 materials: 28 smoothgroups:0) ...
    ( 18.6672 sec |  18.6405 sec) Done.
  ( 18.6673 sec |  18.6671 sec) Finished importing: '/data/jjgd2e3f46f7cc1e8ba69a5e14689f7b974/models/model.obj'
Progress: 100.00%

However, if the same object with ~12k vertices is imported first I get something as follow:

(  0.0001 sec |   0.0001 sec) Importing OBJ '/data/jjgd2e3f46f7cc1e8ba69a5e14689f7b974/models/model.obj'...
  (  0.0025 sec |   0.0023 sec) Parsing OBJ file...
    (  0.5541 sec |   0.5516 sec) Done, loading materials and images...
    (  0.5572 sec |   0.5547 sec) Done, building geometries (verts:12668 faces:43914 materials: 28 smoothgroups:0) ...
    (  1.0660 sec |   1.0635 sec) Done.
  (  1.0663 sec |   1.0662 sec) Finished importing: '/data/jjgd2e3f46f7cc1e8ba69a5e14689f7b974/models/model.obj'
Progress: 100.00%

Here's my code:

#blenderClass.py
import bpy, math, timeit
import numpy as np

class Blender(object):
    def __init__(self):
        self.bpy = bpy
        self.scene = self.bpy.context.scene
        self.scene.render.use_sequencer = False

        # Some memory management
        self.scene.render.use_free_image_textures = True
        self.bpy.context.user_preferences.edit.undo_steps = 0
        self.bpy.context.user_preferences.edit.undo_memory_limit = 60
        self.bpy.context.user_preferences.edit.use_global_undo = False

    def setupScene(self):
        self.removeCamera()
        self.removeMesh()
        self.bpy.ops.object.camera_add(location=tuple(1, -0.5, 0.3))
        self.pointObjTo(self.scene.objects.active, (0.0, 0.0, 0.0)) # My objects are all centered on (0, 0, 0)

    def render(self, objPath):
        self.bpy.ops.import_scene.obj(filepath=objPath)
        self.removeMesh()
        self.removeDataBlocks()


    def removeDataBlocks(self, removeAll=False):
        # Removes unlinked data blocks and prevents memory leakage

        for block in self.bpy.data.meshes:
            if block.users == 0:
                self.bpy.data.meshes.remove(block)

        for block in self.bpy.data.materials:
            if block.users == 0:
                self.bpy.data.materials.remove(block)

        for block in self.bpy.data.textures:
            if block.users == 0:
                self.bpy.data.textures.remove(block)

        for block in self.bpy.data.images:
            if block.users == 0:
                self.bpy.data.images.remove(block)


    def removeMesh(self, layer = -1):
        for obj in self.scene.objects:
            if obj.type == 'MESH':
                obj.select = True
            else:
                obj.select = False
        self.bpy.ops.object.delete()


    def removeCamera(self):
        for obj in self.scene.objects:
            if obj.type == 'CAMERA':
                obj.select = True
            else:
                obj.select = False
        self.bpy.ops.object.delete()


    def pointObjTo(self, obj, xyzTarget):
        # This function operates directly on the input object (obj)
        from mathutils import Vector
        xyzTarget = Vector(xyzTarget)
        direction = xyzTarget - obj.location
        rot_quat = direction.to_track_quat('-Z', 'Y')
        obj.rotation_euler = rot_quat.to_euler()

And this is how I run the code:

#main.py
import blenderClass import Blender
blender = Blender()
blender.setupScene()

objPaths = ['obj1.obj', 'obj2.obj', 'obj3.obj', 'obj4.obj']

for objPath in objPaths:
    blender.render(objPath)

Unfortunately I cannot monitor the system resources in a precise way (I am running this on a server) but I'm afraid that the import function does not let go of some of the resources or somehow memory is getting filled up. I tried importing many shapes into Blender in my Desktop computer and executed the data block removal function after removing the meshes manually. Doing that reduces the memory consumption to something around 10MBs, even if it was 400MBs when I import many 3D shapes.

I'm not sure if this is related but the way I'm calling Blender functions is not by calling it in the background. Instead, I have manually complied Blender 2.79 as a Python module and import its API through import bpy in the Python installed on my machine.

Although I am fairly certain that removing the data blocks frees up the memory but I have also tried using Python's garbage collector and it does not help.

Does anyone know what I can be doing wrong? I am very confused ...

$\endgroup$
  • $\begingroup$ Have you tried to run linux top during running of your script. Could be that your server is limited on memory and alloc/dealloc will then be slow? Also is the overall performance slower, or just a specific part (you can always run your script with profiling enabled to collect more statistics. You can also test with having a custom startup.blend and use the reload startup file operator. $\endgroup$ – J. Bakker Mar 5 '18 at 7:29
  • $\begingroup$ The server has 256 GBs of memory and I'm sure it won't run out of memory that quickly. Running top shows that there is a lot of memory available all the time. What do you mean by having a custome startup.blend file? $\endgroup$ – Amir Mar 5 '18 at 7:57
  • $\begingroup$ Running your script to import 48 objects of 126k verts (4 uniques obj files) using both blender GUI and as a module, I see stable times and memory usage. One thing to try is starting a clean instance for each import. subprocess.call(['python3','myscript.py','obj1.obj']) - using sys.argv to get the filename - instead of repeated import and delete all in one instance. $\endgroup$ – sambler Mar 6 '18 at 3:04
3
$\begingroup$

One problem with your removeDataBlocks() code is that you're modifying the same lists you're iterating over. This can cause items in the list to be skipped, or could, in severe cases, even crash Blender. I would rather do something like this:

to_remove = [block for block in self.bpy.data.meshes if block.users == 0]
for block in to_remove:
    self.bpy.data.meshes.remove(block)

You can also simplify this:

        if obj.type == 'MESH':
            obj.select = True
        else:
            obj.select = False

to just obj.select = obj.type == 'MESH'.

I also don't understand why you would do self.bpy = bpy, since bpy is a module that you can always use directly anyway.

You also don't have to delete and recreate the camera; it will be enough to place it at the desired location & orientation.

Finally, you removeDataBlocks() doesn't remove from bpy.data.objects, so that collection could be filling up and causing the slowdown.

$\endgroup$
  • $\begingroup$ Yes there is no need to do self.bpy = bpy or remove the old camera and add new ones but for I have to do this for some reason; this is a very simplified version of my code. The way I checked the previous code for removing data blocks was to make sure there is no data blocks with users > 0. So I'm not sure this was the best check for removing data blocks or not. Anyways, I made the changes you suggested for removing data blocks but the problem persists still. Did you try the code as well? $\endgroup$ – Amir Mar 4 '18 at 22:00
  • $\begingroup$ I didn't try, because you didn't provide OBJ files to test with. $\endgroup$ – dr. Sybren Mar 4 '18 at 23:03
  • $\begingroup$ Maybe an easy solution would be to use primitive shapes such as spheres, cubes etc and subdivide them so that they will have lots of vertices and consist of different meshes and store them as obj. Do you think you can try that before I think of sharing my obj files? I think something around 10-15 should work. Things start getting slow pretty quickly after importing the 3rd or 4th object for me. Also, do you think my code should work normally? If so, do you think it is possible that compiling Blender as a Python module makes things behave weird? $\endgroup$ – Amir Mar 5 '18 at 5:51
  • $\begingroup$ Could you also elaborate on what you mean by removing from bpy.data.objects? For now I can share one of the meshes I use. You can download it from here $\endgroup$ – Amir Mar 5 '18 at 5:52
  • 3
    $\begingroup$ I've written this on my mobile phone, without Blender at hand, spending time trying to help you out. On the other hand, you don't provide a set of OBJ files to make it simple to actually reproduce your problem. Furthermore, I think all points I've raised are still valid, even though they may not fully solve your problem. I'm in no obligation what so ever to help you, so show a little gratitude instead of vending frustration. $\endgroup$ – dr. Sybren Mar 5 '18 at 16:18
-1
$\begingroup$

Although strange, but it looks like the import function does some stuff that causes the slow down in the long run if I do not import the object with the flag on. So replacing the obj import line with the following would resolve the issue:

self.bpy.ops.import_scene.obj(filepath=objPath, split_mode="OFF")

However I would prefer to input the disjoint object but this solves the importing slow down issue for now. Thanks to doublebishop on blenderartists.

$\endgroup$
  • 2
    $\begingroup$ How many objects are in each obj file? Blender works better with one high poly object than 1000 low poy objs. $\endgroup$ – sambler Mar 6 '18 at 3:08
  • $\begingroup$ @sambler Most of my obj files have many objects in them, something in order of hundred(s), if not more sometimes. Although strange but importing them as one object works perfectly fine. I don't know what does the import function do that things become pretty slow if the obj file has many objects. $\endgroup$ – Amir Mar 6 '18 at 4:02
  • $\begingroup$ that's not the importer - blender gets slower with more objects. Importing hundreds of objs into one object will bypass that. Remove the delete step in your example and see what speeds you get when you add all the new objects to the same scene. $\endgroup$ – sambler Mar 6 '18 at 12:16
  • $\begingroup$ @sambler But the thing is I process each of the imported meshes (consisting of many objects) and then join all of the meshes. What could be still slowing down things? As you can see, I also remove all of the data blocks. So, to my knowledge, there is nothing left to remove. So I'm confused why importing many objects and joining them all into one still causing slowness. $\endgroup$ – Amir Mar 6 '18 at 17:04
  • $\begingroup$ For a single obj file how many objects are in it? If the one file has 1000 separate objects then creating the multiple objects will be slow. By using split_mode=off blender creates one object and joins all of the mesh data into one as it imports, so it is only creating one high poly object per file, this will be faster than importing 1000 low poly objects from one file. $\endgroup$ – sambler Mar 7 '18 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.