I think you have already discovered the problem with a plugin approach. They will look for unlinked media (images not associated with a post) and remove those files. As you have said, you are using custom fields and you may also have uploaded images inside your post content.
Sadly the only option I can think of is to use a find command to get a list of all of your wp-content/uploads and then reduce that list by removing all images in use. As a starting point, you could try the WP-CLI database search command.
You’d want to allow for the different generated image sizes given that a thumbnail or medium image may be used, but you’d want to keep the original + all scaled variants.
It goes without saying that you should have a full backup of your database and files before running anything like this!
I’d be interested if someone can provide a more elegant and robust solution, but this is probably how I’d approach the problem.