The speed of Freestyle is determined by the complexity of your models and render resolution. It runs on CPU only, and cannot use multiple cores. Other than making your models simpler and topology cleaner, there is nothing in particular you can do. If your models are high detail sculpts, re-topologize them (but I assume they are not from your description).
It is possible that certain strange topology issues can bug it out and make it take a long time. And if you are getting very serious slow down (more than a couple minutes render time), it's possible you are running out of RAM and going into pagefile. If you share a screenshot of your models, your render times, and your RAM amount we can determine more.
Alternatively, you can use less Freestyle and mix it with other lining techniques in order to speed up the process. For example, if you replace your Freestyle Silhouette/Contour lines with solidify shell method of lining, and only use freestyle on the internal lines (note that you may need to use layers and have 2 version of the model. One with the shell, and the other without that FS runs on): https://blendernpr.org/solidify-modifier-contouroutline/
There are also compositor based edge nodes: http://blendernpr.org/edge-node-v1-2-4-july-2015/