14
$\begingroup$

I am using Blender 2.80 on my local machine and the machines I am connecting to via ssh. My local machine is Ubuntu 18.04. I hope to use Blender to programmatically generate images as part of my code base, and to do this via ssh.

I installed Blender by taking the tar.bz2 file, and then running

tar -xvjf blender-2.80-d93a7290e506-linux-glibc224-x86_64.tar.bz2

and then in my ~/.bashrc, I have:

export PATH=${HOME}/blender-2.80-d93a7290e506-linux-glibc224-x86_64:$PATH

I have the following minimal working example:

#!/usr/bin/env blender --python
import bpy
bpy.ops.render.render()
bpy.data.images['Render Result'].save_render(filepath='example.png')

Suppose that file is called example-make-image.py. I call it via the following command:

$ blender --background --python example-make-image.py
Blender 2.80 (sub 74) (hash d93a7290e506 built 2019-06-11 23:26:36)
Read prefs: /home/user/.config/blender/2.80/config/userpref.blend
found bundled python: /home/user/blender-2.80-d93a7290e506-linux-glibc224-x86_64/2.80/python
Warning: 'OBJ_SEQUENCE_properties' doesn't contain '_PT_' with prefix & suffix
register_class(...):
Warning: 'OBJ_SEQUENCE_properties' doesn't contain '_PT_' with prefix & suffix
0  meshes freed
Fra:1 Mem:88.26M (0.00M, Peak 88.43M) | Time:00:00.08 | Syncing Cube
Fra:1 Mem:88.63M (0.00M, Peak 88.98M) | Time:00:00.08 | Syncing Light
Fra:1 Mem:88.64M (0.00M, Peak 88.98M) | Time:00:00.08 | Syncing Camera
Fra:1 Mem:88.64M (0.00M, Peak 88.98M) | Time:00:00.10 | Rendering 1 / 64 samples
Fra:1 Mem:88.64M (0.00M, Peak 88.98M) | Time:00:00.15 | Rendering 26 / 64 samples
Fra:1 Mem:88.64M (0.00M, Peak 88.98M) | Time:00:00.17 | Rendering 51 / 64 samples
Fra:1 Mem:88.64M (0.00M, Peak 88.98M) | Time:00:00.18 | Rendering 64 / 64 samples
Fra:1 Mem:48.48M (0.00M, Peak 88.98M) | Time:00:00.22 | Sce: Scene Ve:0 Fa:0 La:0

Blender quit

This will correctly generate an image example.png which I can open to see. It's a gray-ish image of the standard starting cube.

Now I would like to do the same thing via ssh over other machines. Here's a naive way of doing it. I just ssh normally into my machine with no special arguments, but I get this:

$ blender --background --python example-make-image.py
Blender 2.80 (sub 74) (hash d93a7290e506 built 2019-06-11 23:26:36)
found bundled python: /home/user/blender-2.80-d93a7290e506-linux-glibc224-x86_64/2.80/python
Unable to open a display
Aborted (core dumped)

Fair enough. I follow these instructions to enable me to run graphics applications over ssh. I connect via ssh -X user@machine-address, and I can confirm by running xterm & that I can open png images remotely. But when I try to run the same blender script above, I get:

$ blender --background --python example-make-image.py
Blender 2.80 (sub 74) (hash d93a7290e506 built 2019-06-11 23:26:36)
debug1: client_input_channel_open: ctype x11 rchan 3 win 65536 max 16384
debug1: client_request_x11: request from 127.0.0.1 59644
debug1: channel 1: new [x11]
debug1: confirm x11
debug1: channel 1: FORCE input drain
debug1: channel 1: free: x11, nchannels 2
debug1: client_input_channel_open: ctype x11 rchan 3 win 65536 max 16384
debug1: client_request_x11: request from 127.0.0.1 59646
debug1: channel 1: new [x11]
debug1: confirm x11
debug1: channel 1: FORCE input drain
debug1: channel 1: free: x11, nchannels 2
ALSA lib pcm_dmix.c:1029:(snd_pcm_dmix_open) unable to open slave
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': Device or resource busy
found bundled python: /home/user/blender-2.80-d93a7290e506-linux-glibc224-x86_64/2.80/python
debug1: client_input_channel_open: ctype x11 rchan 3 win 65536 max 16384
debug1: client_request_x11: request from 127.0.0.1 59648
debug1: channel 1: new [x11]
debug1: confirm x11
Received X11 Error:
    error code:   2
    request code: 154
    minor code:   34
    error text:   BadValue (integer parameter out of range for operation)
Received X11 Error:
    error code:   2
    request code: 154
    minor code:   34
    error text:   BadValue (integer parameter out of range for operation)
Received X11 Error:
    error code:   2
    request code: 154
    minor code:   34
    error text:   BadValue (integer parameter out of range for operation)
Received X11 Error:
    error code:   2
    request code: 154
    minor code:   34
    error text:   BadValue (integer parameter out of range for operation)
Received X11 Error:
    error code:   2
    request code: 154
    minor code:   34
    error text:   BadValue (integer parameter out of range for operation)
Received X11 Error:
    error code:   2
    request code: 154
    minor code:   34
    error text:   BadValue (integer parameter out of range for operation)
Received X11 Error:
    error code:   2
    request code: 154
    minor code:   34
    error text:   BadValue (integer parameter out of range for operation)
Writing: /tmp/blender.crash.txt
debug1: channel 1: FORCE input drain
Segmentation fault (core dumped)
debug1: channel 1: free: x11, nchannels 2

and no image is created.

Hopefully this explains my issue clearly. A few other things:

Is this error reproducible by others?

Let me know if any more information would be helpful.

UPDATE 1: I've reproduced the setup again, and here is the output in /tmp/blender.crash.txt:

# Blender 2.80 (sub 74), Commit date: 2019-06-11 21:45, Hash d93a7290e506

# backtrace
blender(BLI_system_backtrace+0x1d) [0x13835ad]
blender() [0x1169ef9]
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7febe4c9b4b0]
blender(GHOST_ActivateOpenGLContext+0) [0x2f37960]
blender(DRW_opengl_context_create+0x3d) [0x2a38b2d]
blender(WM_init_opengl+0x20) [0x1541cc0]
blender(DRW_render_to_image+0x2fb) [0x2a39b2b]
blender(RE_engine_render+0x311) [0x1412751]
blender() [0x1413e21]
blender() [0x1416bc8]
blender(RE_RenderFrame+0x114) [0x1417554]
blender() [0x3ea66bb]
blender() [0x1532b7b]
blender(WM_operator_call_py+0x60) [0x1533370]
blender() [0x1409f33]
blender(_PyMethodDef_RawFastCallKeywords+0x2b9) [0x176be69]
blender(_PyCFunction_FastCallKeywords+0x25) [0x176bf25]
blender(_PyEval_EvalFrameDefault+0x7dd1) [0x1165011]
blender(_PyEval_EvalCodeWithName+0xae3) [0x1820233]
blender(_PyFunction_FastCallDict+0xb7) [0x176b6f7]
blender(_PyObject_Call_Prepend+0xbd) [0x176c9bd]
blender() [0x17c9d81]
blender(_PyObject_FastCallKeywords+0x104) [0x176c044]
blender(_PyEval_EvalFrameDefault+0x632a) [0x116356a]
blender(_PyEval_EvalCodeWithName+0xae3) [0x1820233]
blender(PyEval_EvalCodeEx+0x3e) [0x182038e]
blender(PyEval_EvalCode+0x1b) [0x18203bb]
blender(PyRun_FileExFlags+0xb2) [0x1853432]
blender() [0x13f5b61]
blender() [0x1168138]
blender(BLI_argsParse+0xd5) [0x133fb95]
blender(main+0x2c9) [0x10d01f9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7febe4c86830]
blender(_start+0x2a) [0x116680a]

UPDATE 2: Following this AskUbuntu question, I am reporting my OpenGL version on the remote machine directly after I log in via ssh -X machine:

$ DISPLAY=:0
$ glxinfo | grep -i opengl
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: TITAN X (Pascal)/PCIe/SSE2
OpenGL core profile version string: 4.5.0 NVIDIA 390.87
OpenGL core profile shading language version string: 4.50 NVIDIA
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6.0 NVIDIA 390.87
OpenGL shading language version string: 4.60 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 390.87
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:
$ 

and

$ glxinfo | grep 'version'
server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4
OpenGL core profile version string: 4.5.0 NVIDIA 390.87
OpenGL core profile shading language version string: 4.50 NVIDIA
OpenGL version string: 4.6.0 NVIDIA 390.87
OpenGL shading language version string: 4.60 NVIDIA
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 390.87
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
    GL_EXT_shader_implicit_conversions, GL_EXT_shader_integer_mix, 
$ 

And it seems to be up to date.

$\endgroup$
13
  • $\begingroup$ Do you try to render on the GPU or CPU? $\endgroup$ – Robert Gützkow Jun 29 '19 at 11:04
  • $\begingroup$ To me it looks like you're connecting to a VM which doesn't have the appropriate permissions set in order to use the GPU via ssh. Additionally the seems to be trouble with OpenGL / GLX which why these "Received X11 Error" appear. $\endgroup$ – Robert Gützkow Jun 29 '19 at 11:57
  • $\begingroup$ @gtzkw I am not sure which of the GPU/CPU I am using to render. The machine that I am connecting to via ssh is another personal workstation that I use with an NVIDIA Titan X GPU (state of the art when it was released in mid 2016, I think). So I have two machines here and am sudo on all of these. I will look and see how to get more permissions. $\endgroup$ – ComputerScientist Jun 29 '19 at 20:18
  • $\begingroup$ As far as I'm aware using the GPU requires an X-session. In your example it crashes for some reason. Perhaps the /tmp/blender.crash.txt can tell us more. $\endgroup$ – Robert Gützkow Jun 29 '19 at 20:53
  • $\begingroup$ @gtzkw Sounds good, added the /tmp/blender.crash.txt file. $\endgroup$ – ComputerScientist Jun 29 '19 at 21:20
8
$\begingroup$

I figured out how to answer my question. The main source that I relied on was this wonderful answer (and then this follow-up) in the Blender devtalk thread that I referenced to in my original question. (Those answers were posted after I made this question here.) Here's my answer which somewhat cleans up their explanations, but credit goes to those two who answered on devtalk.

Here are the steps that I followed. Here, I assume that the remote machine has an NVIDIA GPU, is running Ubuntu 18.04, and has Blender 2.82a installed, with the blender command runnable.

(1) I ssh into my remote machine. I do NOT use X forwarding, so I do not use ssh -X.

(2) Running glxinfo returns this error:

$ glxinfo
Error: unable to open display

If you don't have glxinfo, run sudo apt install mesa-utils.

(3) To fix the "unable to open display" error, first run the following to see available GPU(s):

$ nvidia-xconfig --query-gpu-info
Number of GPUs: 1

GPU #0:
  Name      : TITAN X (Pascal)
  UUID      : GPU-9adbb93a-e6c6-b54c-651c-66d1414a6b82
  PCI BusID : PCI:1:0:0

You may get other outputs for additional GPUs (for multi-GPU machines) and for monitors if there are attached monitors. For now we don't worry about those. We just pick the GPU we want, and look at the PCI BusID value.

(4) Next, we run this, where I insert the PCI BusID into the --busid argument:

$ sudo nvidia-xconfig --busid=PCI:1:0:0 --use-display-device=none --virtual=1280x1024
[sudo] password for user:

Using X configuration file: "/etc/X11/xorg.conf".
Backed up file '/etc/X11/xorg.conf' as '/etc/X11/xorg.conf.backup'
New X configuration file written to '/etc/X11/xorg.conf'

According to devtalk, this will "create a virtual x screen" and "create[s] the xorg.conf file. Xserver will read this file and setup the xorg server accordingly".

(5) Now, run sudo Xorg :1. If you get an error like I originally had:

$ sudo Xorg :1
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
(EE)
Fatal server error:
(EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE)
(EE)
Please consult the The X.Org Foundation support
     at http://wiki.x.org
 for help.
(EE) Please also check the log file at "/var/log/Xorg.1.log" for additional information.
(EE)
(EE) Server terminated with error (1). Closing log file.

Then run ps -C Xorg to see what Xorg programs are already running. Then use sudo kill -9 [pid].

(6) A working command produces this output for me:

$ sudo Xorg :1

X.Org X Server 1.20.4
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-148-generic x86_64 Ubuntu
Current Operating System: Linux takeshi 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.3.0-40-generic root=UUID=99a59366-4621-4034-b183-bfaf2a8af7eb ro quiet splash vt.handoff=1
Build Date: 02 May 2019  08:06:54AM
xorg-server-hwe-18.04 2:1.20.4-1ubuntu3~18.04.1 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.34.0
    Before reporting problems, check http://wiki.x.org
    to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
    (++) from command line, (!!) notice, (II) informational,
    (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.1.log", Time: Mon Apr 27 12:06:11 2020
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"

Now, this will be running in one shell. Using CTRL+C will not work to kill it. For your subsequent work, you can use a separate shell.

(7) For this, I decided to do another ssh command to enter my remote machine. This step is critical: export DISPLAY=:1. This will set the display variable to be what the Xorg command did in the other shell.

(8) As a sanity check, make sure that glxinfo returns expected output:

$ glxinfo | grep -e render -e NVIDIA
direct rendering: Yes
server glx vendor string: NVIDIA Corporation
client glx vendor string: NVIDIA Corporation
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: TITAN X (Pascal)/PCIe/SSE2
OpenGL core profile version string: 4.6.0 NVIDIA 430.50
OpenGL core profile shading language version string: 4.60 NVIDIA
    GL_ARB_conditional_render_inverted, GL_ARB_conservative_depth,
    GL_NVX_conditional_render, GL_NVX_gpu_memory_info, GL_NVX_nvenc_interop,
    GL_NV_command_list, GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_path_rendering_shared_edge,
    GL_NV_shader_thread_shuffle, GL_NV_stereo_view_rendering,
OpenGL version string: 4.6.0 NVIDIA 430.50
OpenGL shading language version string: 4.60 NVIDIA
    GL_ARB_compute_variable_group_size, GL_ARB_conditional_render_inverted,
    GL_NVX_conditional_render, GL_NVX_gpu_memory_info, GL_NVX_nvenc_interop,
    GL_NV_command_list, GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_path_rendering_shared_edge,
    GL_NV_shader_thread_shuffle, GL_NV_stereo_view_rendering,
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 430.50
    GL_EXT_multisampled_render_to_texture,
    GL_EXT_multisampled_render_to_texture2, GL_EXT_occlusion_query_boolean,
    GL_EXT_render_snorm, GL_EXT_robustness, GL_EXT_sRGB,
    GL_NV_clip_space_w_scaling, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_path_rendering_shared_edge,
    GL_NV_shadow_samplers_cube, GL_NV_stereo_view_rendering,
    GL_OES_element_index_uint, GL_OES_fbo_render_mipmap,
    GL_OVR_multiview_multisampled_render_to_texture

(9) Finally, I run my ultimate test: blender -b -E BLENDER_EEVEE -f 1 to see if the rendering works. If all goes well, you'll see this output:

$ blender -b -E BLENDER_EEVEE -f 1
Blender 2.82 (sub 7) (hash 375c7dc4caf4 built 2020-03-12 05:30:40)
found bundled python: /home/user/blender-2.82a-linux64/2.82/python
Fra:1 Mem:88.40M (0.00M, Peak 88.68M) | Time:00:00.32 | Syncing Cube
Fra:1 Mem:88.72M (0.00M, Peak 89.01M) | Time:00:00.32 | Syncing Light
Fra:1 Mem:88.72M (0.00M, Peak 89.01M) | Time:00:00.32 | Syncing Camera
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.33 | Rendering 1 / 64 samples
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.37 | Rendering 26 / 64 samples
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.39 | Rendering 51 / 64 samples
Fra:1 Mem:88.73M (0.00M, Peak 89.01M) | Time:00:00.40 | Rendering 64 / 64 samples
Fra:1 Mem:48.47M (0.00M, Peak 89.01M) | Time:00:00.43 | Sce: Scene Ve:0 Fa:0 La:0
Saved: '/tmp/0001.png'
 Time: 00:00.75 (Saving: 00:00.31)


Blender quit

I am able to secure-copy (using scp) that image onto my local machine, and I can confirm that it looks correct.

(10) Once I'm done, the last thing I do is run nvidia-smi to see what processes are there:

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     22948      G   /usr/lib/xorg/Xorg                            14MiB |
|    0     22976      G   /usr/bin/gnome-shell                           6MiB |
|    0     23332      G   /usr/lib/xorg/Xorg                            10MiB |
|    0     30462      C   python                                       847MiB |
+-----------------------------------------------------------------------------+

Then I can run the usual sudo kill -9 [pid] where the process ID comes from above. I think one of the two Xorg commands will constantly restart itself so it may not be easy to kill, but I will need to do more investigation.

Final comments: thanks to @sambler in the comments, I now know that to test the rendering, it is easier to use these one-line commands:

blender -b -E BLENDER_WORKBENCH -f 1
blender -b -E BLENDER_EEVEE -f 1
blender -b -E CYCLES -f 1 

rather than the python script that I was using in my original question. These use all three of Blender's available rendering engines. The last command, using CYCLES, will work remotely without the bells-and-whistles here, while the first two will not. But the downside is that CYCLES is much slower than the other rendering engines.

I did another test where I enter GNU screen (via screen command) and ran the same one-line blender testing command above. It seems to work successfully, suggesting that one can run in screen and then log out of the machine, and have images repeatedly be generated.

You can also run the Xorg command in a screen shell so that you can exit it, and the command will just run permanently until you turn it off. I was able to do this and run the Blender tests successfully. This is probably my preferred setup since it doesn't require me to explicitly leave a "tab" open (e.g., a tab in iTerm2, which I use to ssh and to program).

Finally, for faster iterations, you can combine the display and the blender commands:

DISPLAY=:1  blender -b -E BLENDER_EEVEE -f 1

and this should also work.

Update 1: sometimes, step (6) does not work. I've noticed that if you log into a remote machine that does not already have an X-server running (which I think we can tell by running nvidia-smi and checking that there exists an Xorg process running) then you might get this error:

$ sudo Xorg :1

X.Org X Server 1.20.4
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-148-generic x86_64 Ubuntu
Current Operating System: Linux mason 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:12:28 UTC 2019 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-23-generic root=UUID=877a0d47-b6cc-4687-8778-3a947fc7cb04 ro quiet splash vt.handoff=1
Build Date: 02 May 2019  08:06:54AM
xorg-server-hwe-18.04 2:1.20.4-1ubuntu3~18.04.1 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.34.0
    Before reporting problems, check http://wiki.x.org
    to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
    (++) from command line, (!!) notice, (II) informational,
    (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.1.log", Time: Wed Apr 29 10:11:48 2020
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
(EE)
Fatal server error:
(EE) no screens found(EE)
(EE)
Please consult the The X.Org Foundation support
     at http://wiki.x.org
 for help.
(EE) Please also check the log file at "/var/log/Xorg.1.log" for additional information.
(EE)
(EE) Server terminated with error (1). Closing log file.

In this case, starting up a normal X server should work. However, I have yet to get this working. The above error message with a machine that has GPUs, but does not have an attached monitor, whereas I got mine working from another machine that did have a monitor. I think the fix is to consult these references (source 1, source 2, source 3).

Update 2 (July 21, 2020): I tried running:

sudo nvidia-xconfig --busid=PCI:1:0:0 --virtual=1280x1024  --use-display-device="DFP-0"

and I have been able to fix the above error with that. I would suggest starting with the above if that works.

$\endgroup$
1
  • $\begingroup$ How would you go about it without gpu rendering?. I use a general purpose AWS EC2 T3 instance and came upon "Unable to open a display" while trying to render in the background. $\endgroup$ – johnconnor Jan 29 at 0:39
0
$\begingroup$

To answer a few of your questions: ssh -x is indeed forwarding X-11 (X-Window) traffic from your remote machine to your local one. It assumes you have a local display. If you were to ssh to machine A and then from there ssh -x to machine B, you would not have a display. It might be best to start on your machine with a display and ssh to machine B directly.

Blender is primarily a GUI application and I'm not too familiar with running it without displaying graphics. However, it seems --background should be doing that. But, clearly it is not keeping it entirely from running without communicating to X-11. So, there may be something to add to the python program to get it to exit blender when completed, instead of maybe starting up the GUI. Like:

import sys
sys.exit(0)

Another option: Do you need to run blender remotely or can you do the processing on one machine and copy the results (scp) to other machines?

$\endgroup$
2
  • $\begingroup$ Thanks. The issue is that I need to run many, many copies of Blender in parallel. My work relates to imitation learning and we have to run many simulations to get lots of data. That unfortunately means I need to use servers that have many CPUs, but which are headless. $\endgroup$ – ComputerScientist Apr 13 '20 at 17:45
  • $\begingroup$ Well, here is the next thing I would try. Connect via ssh but without the -x, so X11 should stay on localhost (the remote host). You could try: DISPLAY=localhost:0.0 to force it. It may result in a permission error, and there is probably a way around that, but I think it would be more fun to run a headless X11 server on the remote machine. Take a look at this answer to another question for an option to try: stackoverflow.com/a/834801/238074 . It uses an X11 server that just tosses all the output. Should be good enough to fool blender. $\endgroup$ – Kevin Buchs Apr 16 '20 at 13:03
0
$\begingroup$

virtual x screen tool:xvfb work for me

sudo apt-get install -y xvfb

xvfb-run --auto-servernum --server-args="-screen 0 1600x1024x16" blender -b -o /tmp/ -E BLENDER_EEVEE -f 1
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.