RedirectMatch 301 ^/author/$ ^/auteur/$1
This will only redirect /author/, but the target URL is invalid (with a ^ prefix). The target URL is an ordinary string (that accepts placeholders), not a regex.
However, since WordPress already uses mod_rewrite directives for the front controller. You should also use mod_rewrite for this redirect (as opposed to a mod_alias RedirectMatch) in order to avoid potential conflicts. (The different Apache modules: mod_alias, mod_rewrite, etc. run independently at different times during the request, regardless of the apparent order of the directives in .htaccess – this can make any potential conflicts rather confusing to debug.)
Try something like the following instead before the existing WordPress directives:
RewriteRule ^author/(.*) /auteur/$1 [R=302,L]
This will redirect /author/ to /auteur/ and /author/<anything> to /auteur/<anything>.
This is currently a 302 (temporary) redirect. Change this to a 301 (permanent) only when you are sure it’s working OK. 301 redirects are cached hard by the browser so can make testing problematic.
Make sure your browser cache is clear before testing.