As far as I know, the few files that you are talking about are always going to be the same, that means you can add exceptions to your .htaccess and “let them go through the cracks”.
With an example of code from what you wrote so far may help us direct you on how to proceed to do this.
Here is an example of how this is generally done although that uses old code (Allow/Deny) which I don’t recommend. It shows you how to change the Deny into an Allow using the <FilesMatch> tag.