First, everyone doesn’t say that. Everyone says to use admin-ajax.php for ajax, which is it’s meant for, not form submissions precisely. It’s not as popular and well covered, but there is actually different endpoint for form submission — admin-post.php.
The reasons are, however, same. What you cannot do in your plugin is reliably load WordPress core. When WordPress core loads plugins it composes coherent reliable environment, but it doesn’t work the other way around.
The point of using core–provided endpoints is precisely to rely on that environment.