As you’ve stated, the B flag is required in this case. But this should be [B,L], not [BNC] as you’ve quoted a couple of times?
Not sure where you got [BNC] from, but that’s wholly invalid and would break an Apache server (500 Internal Server Error response). If you don’t see an error, using this invalid flag, then it’s possible you are on a LiteSpeed server, which quietly ignores the error and the directive does not run, allowing unrestricted access to the file (which seems to be what you are seeing here).
The B flag is required in order to re-encode the & (as %26) as captured from the requested decoded-URL before passing this to the query string, so that you pass the entire “filename” and not just the part before the &.
For example, if you request /wp-content/uploads/abc%26def.pdf and do not use the B flag then the resulting request will be:
checkloggedin.php?file=abc&def
That’s now effectively two URL parameters: file=abc and def=
With the B flag, the captured backreference is re-encoded and becomes:
checkloggedin.php?file=abc%26def
PHP/WordPress then decodes this to abc&def – the complete filename.
So, the complete rule should be:
RewriteCond %{REQUEST_FILENAME} -s
RewriteRule ^wp-content/uploads/(.*) checkloggedin.php?file=$1 [B,L]
The QSA flag is not required, unless you are passing additional query string parameters in your initial file request? The trailing $ on the RewriteRule pattern is not required either, since regex is greedy by default.
If this still does not work then the problem is with the checkloggedin.php script. But from what you say, this script is not even being called when using BNC – which is why I think you are on a LiteSpeed server (not Apache).
UPDATE:
An alternative method is to pass the filename as path-info instead of as a URL parameter in the query string. You are then not prone to URL-encoding issues associated with & (this is not a special character in the URL-path).
So, instead of the above rule, you would use the following instead in .htaccess:
RewriteCond %{REQUEST_FILENAME} -s
RewriteRule ^wp-content/uploads/(.*) checkloggedin.php/$1 [L]
The B flag is not required.
The URL-path matched by the RewriteRule pattern is already %-decoded (so it doesn’t matter whether the original request includes & or %26). The same “file” value is passed as path-info (additional pathname information) and is therefore available via the $_SERVER['PATH_INFO'] superglobal, instead of $_GET['file'].
So, in your script you would need to do something like the following instead to populate your $filename variable:
$filename = isset($_SERVER['PATH_INFO']) ? $_SERVER['PATH_INFO'] : null;