How To Allow Only Specific User Agent To Access a URL?

There are at aleast two ways you can block other user agents and allow only a few. This can be done by editing the .htaccess file in the root directory of your WordPress website.

Option 1

Using mod_rewrite, add the code below at the top of your .htaccess file.

<IfModule mod_rewrite.c>

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} !(Google|Bing)
RewriteRule ^(sitemap_index\.xml)$ - [F,L]

</IfModule>

Option 2

This option uses the mod_setenvif module with SetEnvIfNoCase block incase mod_rewrite isn’t available.
Add the code below at the top of your .htaccess file.

<IfModule mod_setenvif.c>

SetEnvIfNoCase User-Agent .*google.* allowed_user_agents
SetEnvIfNoCase User-Agent .*bing.* allowed_user_agents

Order Allow,Deny
<FilesMatch "^(sitemap_index\.xml)$">
Deny from all
Allow from env=allowed_user_agents
</FilesMatch>

</IfModule>

You could also use robots.txt file, but I haven’t tested this one yet.