TL;DNR: There is no best but Client/Server is used most often so I would recommend that approach.
There is no single best way, because all of the approaches have strengths and weaknesses and the one that works best in a given situation will depend on which weaknesses your application can tolerate.
Since you don't like the first approach I'll only comment on the other two.
Client/Server approaches are the most flexible and allow you to have the client and server on different machines; but they rely on networking libraries that can be a bit tricky to use and hard to debug in weird corner cases. Still, they're the type of approach most commonly used in these situations.
File based approaches are subject to various problems with timing, OS buffering of file contents, and conflicts when you try to use the same technique from two instances of Blender at the same time; but they take very little code to implement.
Two other approaches that are more complex and probably over-engineered for your application are
Shared memory based approaches. These are a hybrid of client/server and file based approaches in that they tend to use mailboxes. They're a bit more robust than client/server but also a bit more complex.
Blender as a module. Complete overkill for your use.
That said, I would recommend client/server because there are plenty of examples; it's not that difficult to adapt existing approaches; and there's a community where you can get support.
