I’ve struggled with the same thing pretty much always. There’s a plugin called File Un-Attach, and a Unattach and Re-attach Media Attachments. See if they work for you.
There’s also 5 year old Trac ticket on this, but it seems to be under conversation still.