You're misinterpreting what you see. An
Image Texture node is not an image, but uses an
Image. These are two distinct classes in Blender. When reading Image somewhere in Blender, think of an instance of the class
Image, not an image file (like .png) on disk. I'll refer to Image as "image object".
When you click either the New Image or the Open Image button in an
Image Texture node, Blender will always create a new image object and connect that as a data block for use in the node.
Note that the name left to the buttons is just a name taken from the filename when you chose Open Image or your selfchosen name (default "Untitled") with New Image. In that regard, Open Image is slightly confusing because it really means New Image from File.
Have a look at the following image (haha), I paired a Shader Editor and an Image Editor and opened the side bars (the little
< icon in the top right corner):
I also paired identical menus with the colored boxes. As said, the red boxes are for creating new image objects (or switching to another existing one with the icon left to the name) but they are not for changing settings of existing image objects. This is what the yellow boxed menus are for. They show the settings of an existing image object and this is completely independent from an
Image Texture node. Changing these settings is bound to the image object and the
Image Texture node just reflects that (for example the Color Space). Try it: if you change the Color Space in any of the green boxes, it will change too in the others.
If you want to change the actual image (that is the file on disk), you can do that in the blue boxes with the folder button. Note that this folder button looks identical to the button from Open Image, but is actually a different thing, Browse Image. I think that's a bit unfortunate but now you have Reload Image next to it, which makes sense again.
By now, you should see that changing the source file does not change the image object's name or any other setting. You just changed the source file of an existing image object. That is the "replacing" you want.
If you don't like to change an existing image object for that, a quick way to keep all settings is to duplicate the
Image Texture node and make a single-user copy of the image object by clicking the
2 in the duplicated node. In the screenshot you can see that I created that way a new image object
normal1.png.001 (just a name, right) with a different image
.../normal2.png (a real image) and the
Image Texture node settings are unchanged as well as the other image object settings.