Which piece of hardware affects viewport performance most (both playback and general just moving stuff)?
It depends on the scenario.
Situations where the GPU will help more:
- Extremely high poly scenes
- Scenes with complex materials
- Scenes with large textures
- Scenes with advanced procedural textures
The GPU is mainly used to draw the pixels on your screen, so if you have lots to draw, a better GPU will help. A good way to tell if your GPU is the limiting factor in your performance is to check your frame rate when you are only moving your viewpoint, and nothing else.
Situations where the CPU will help more:
- High frame rate animations
- Scenes with lots of ridged bodies or simulations
- Scenes with complex armatures
- Scenes with lots of booleans
- Scenes with geometry nodes
- Complex animations
- Scenes with procedurally generated objects, such as those with displacement modifiers
- Scenes with lots of Multi-resolution/subdivision modifiers
For the most part, the CPU will be your main limiting factor. The CPU must apply all modifiers, animations, deformations, and simulations before finally sending it's data off to the GPU to be displayed. If you are looking for a CPU specifically to speed up the viewport, you should prioritize single threaded performance, as many modifiers are single threaded.