As far as I can see, it is not possible to use a custom 404 ErrorDocument directive in your htaccess file because WordPress’ index.php will always be found when processing the htaccess content. (unless ofcourse index.php got deleted for some reason)
Because index.php is found, it will not trigger the ‘404 not found’ ErrorDocument directive. The ErrorDocument directive will thus be ignored.
That is – if you stick with the htaccess code generated by WordPress (which I strongly recommend).
It is possible to work without the htaccess rules that WordPress generates, but it requires you to write htaccess rules for the processing of permalinks etc yourself.
I did this in a project where I decoupled WordPress almost completely from the frontend, but I can’t recommend doing that for ~98% of the use cases. 🙂