Are you working a obj you created from a photogrammetry ?
IF so, you cannot. Unless you wanted to do that in an earlier part of the process... Photogrammetry takes the mesh and attaches the texture to the entire mesh. So you cannot separate out the texture by color because they are all on the same texture file... you can split up the faces and then assign them a new texture then delete that texture but at that point you're not avoiding the problem of having to delete everything.
When working with Photo scans you just have to except the fact that you will have to do some busy work. There are programs that can auto fix your typology - which may help in selecting methods ( being able to select and delete edge loops), but other than that you will be working at the geometry level which does not see color-only textures hold that data.
This is a really late reply. However, if anyone sees this I hope it helps.