Saving blend-files straight to server nets 0.5% speed of a windows file copy, how to fix?

Question

Some context:

Part of our company (about 3 fulltime people) do ArchViz, we're using Blender 2.79b and are transitioning to Blender 2.8. Tests underneath were done using 2.8. I'm not an artist myself but part of my job is writing scripts, tooling and general problem solving any IT related issue.

For about a year now our designers want to work directly of our network drives (one L:\ share for library one V:\ share for client files). It's to slow to save directly to the network although we've got Gigabit LAN. Using linking doesn't work fluidly either on the network and they want me to add 10GBit to our office.

---

The problem at hand:

A random exterior file I used for testing is 2.2GB. I used a big file since it evens out some potential network delays. I know linking would partly resolve our problems, particularly for interiors, since it reduces our project file size, however this is not what I'm looking for.

We've got gigabit network and some Samsung EVO SSDs. Saving and copying tests of that file resulted in the following:

• Saving the blend file to our local SSD: 4s (4400Mbps)
• Saving the compressed blend file to our local SSD: 49s (n/a)
• Copying the blend file using Windows Explorer to the network share: 23s (765Mbps)
• Saving the blend file directly to the network share: +5000s (<3.5Mbps)
• Saving the compressed blend file directly to the network share: not even attempted

Obviously the way Blender saves files is subpar for network saving. It makes me fear that upgrading to 10Gbit will do absolutely nothing to improve our situation.

---

The questions I have:

• Why is saving files to the network so slow?
• Is there any builtin way to fix this?
• If not, is there any addon that can mitigate this problem?
• If not, are there any resources I should look at to make my own solution?

---

UPDATE 05/09/2019:

I've managed to answer some of the questions myself, hereby my progress:

• According to Bastien Montage my bug report on the Blender bug tracker was invalid as this is something a sysadmin should fix (namely figure out a way to write to local filesystem and then upload to a network share). He also stated:

We do not have control over where you write a file> Improvements are always possible, but that is a feature request then, not a bug, and it has nothing to do on this tracker..

To me the notion that this isn't a bug and a sysadmin should fix it seems like an ignorant approach to the problem that ignores the way professionals (might want to) use Blender.

In all honesty I was very disappointing by his reaction, but it boils down to the fact that they consider this acceptable behavior.

• I proceeded looking into this and found out that his point about 'small chunks' of memory being written is to blame for this. A Stackoverflow article named Python copy larger file too slow seems to corroborate this.

In essence the file write action uses a 16KB buffer size, which works great for local filesystems but doesn't work for remote systems. According to the post this is esspecially true for Windows. I observe almost equally bad problems on Linux machines as well.

Using the python executable that comes bundled with Blender on Windows I tried to copy a new 2GB file to the remote filesystem using both shutil.copy(source, target) and os.system('xcopy "%s" "%s"' % (source, target)) and the results of the first attempt are completely in line with my previous findings. The latter attempt resulted in the exact same speed as copying the file using Windows Explorer.

I don't really manage to find where Blender does its filesystem writeout, but I only looked for it for about 15 minutes. That being said here are the answers to my questions I posted earlier:

Why is saving files to the network so slow?

This seems to be the case because of an exceptionally small buffer used by either Blender or the library used for filesystem writeout. Which library is used and whether it can be changed by changing a simple variable within the Blender sourcecode remains to be determined.

Is there any builtin way to fix this?

At this point, not that I can tell. I think changes to the Blender core are needed.

Quite frankly my experience on the bugtracker does not spark a lot of confidence that this will be resolved soon by the developers themselves, they don't seem to think this is an issue. I initially considered to add the functionality to the core myself but that motivation was gone as soon as Bastien answered me because if its not a bug, and its not on the roadmap you need to convince the developers of the added benefit of it. Getting a new feature in as a first commit to an open-source project almost never works, so I'm passing.

If not, is there any addon that can mitigate this problem?

There should be, I would guess that someone, somewhere already made this. I can't find it however as I came up empty on both Blender Market and Github.

If not, are there any resources I should look at to make my own solution?

It seems like the previous link to the Python Stackoverflow question is a good starting point for fast writing to the network from Python.

A possible workaround would be to make a plugin that does better saving to the network. Since I don't know how Blender does saving internally it could work like this:

• Attach a hook whenever the user uses the save function. (figure out if this still works in 2.80)
• Use GetDriveType to determine whether or not the location you are trying to save to is a network drive.
• If it is, writeout the file to a temporary files location using UserPreferencesFilePaths.temporary_directory.
• Use one of the methods from the first stackoverflow link to copy the file at a greater speed. Either increase the buffer or use xcopy

This would change the save location for blender to the temporary directory however; so next time you save it won't end up on the network. The bpy.data.filepath variable is readonly apparently so we can't change it back to network share after we safe.

We might use a method like suggested here to check where to save on every save attempt but its not a best practice. Another method is to ditch the default save button all together and write our own.

This needs more work still... Any advice welcome!

Answer 1

So, I finally made a blender addon that solves the save problem by first saving the .blend file to the temporary files location, and then copying it to the network share.

Right now it works by remapping your ctrl+s or cmd+s shortcut.

You can find the plugin here:

https://github.com/woutervddn/blender_addon_save_by_proxy/blob/master/README.md