Removing /author/ slug from URL gives 404

You are just doing it wrong. Adding a per user rewrite rule will just bloat the rewrite “table” and slow down the URL parsing process. You need to come up with a generic rewrite rule which will work for all users, or write an alternative url parsing using the ‘do_parse_request filter.

Your core problem is that your URL structure as it is now “collides” with the url structure of posts/categories/etc, and the best way is to just use a constant predictable prefix for your user urls maybe something like /user/{user login}/... otherwise, the only sane option in a site with many users is to do your own parsing.