By default the viewport samples are 32 (at least on my machine). This gives me a rough render fairly quickly when the viewport isn't changed. But are there downsides to increasing this number? Obviously when increasing the render samples, it takes longer to render, but gives you a higher quality image. However, in the viewport it renders progressively and I don't see any reduction in performance after increasing it to 256 or 512. The only downside I can think of would be more wear and tear on my GPU, but is there something I'm not thinking about?
The only disadvantage I see is that it will take longer (as you said) and your machine might run hotter due to the increased time taken rendering, which could after a while decrease the life of your battery, fan, GPU etc. The rate at which your cooling system(s) is running is a good indicator of how much the rendering may be slowing down and/or decreasing the life of your components. Just make sure the rendering not causing the cooling system to run excessively high for long periods of time.