It sounds like you’re embedding the PDF directly into the page using the media library function.
Instead, you can visit the media library directly, copy the permalink of the file you’d like to link to, and use the copied permalink as the hyperlink around for text or the image.
It’s manual, but solves your problem without any bloat or custom fixes.