mod-rewrite rules grabs “broken permalinks”, too

The web server doesn’t know if a request URI matches a valid permalink.

Handling 404 errors happens in WordPress itself. Your theme should have a 404.php file to return properly formatted output.