If you add a new object while in edit mode, the new generated object will be added as part of the active object, but it will still be a separate mesh, means it's vertices, edges and faces are not fused with the object you add it to.
Thus it's possible to select a vertex, edge or face of the newly added object-mesh.
If you press "l", for select linked (it works as well as long as your cursor is close enough to the mesh), to select all connected vertices, edges or faces depending on which edit mode you are in.
After you selected the mesh you want to delete, press "x" and select Vertices, as that will delete all basis for the mesh, deleting edges or faces would still leave the vertices behind and would eventually need another clean up.
Should you decide not to delete the mesh/geometry you selected, press "p" for the separate function and go with "Selection", that will generate another object with the selected and now split of geometry.