How do I skip wordpress’s 404 handling?

Native WordPress rules are designed to ignore any existing files and directories, including arbitrary PHP scripts. This is literally what this part of directives mean:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

So under normal circumstances WP just shouldn’t be involved with your request in any way. You may have some other rewrite rules interfering, possibly from a different place in web server configuration.