Why does the header set X-Robots-Tag apply to all pages?

The E flag is being used for LiteSpeed as opposed to ENV for Apache.

Where did you get this from? The LiteSpeed docs appear to show that LiteSpeed is the same as Apache in this respect and require env=, not E=, on the Header directive in order to conditionally set the HTTP response header based on the presence of an environment variable.

Having tested this myself on a LiteSpeed server I can confirm that this does appear to be the case. And this would seem to be the problem with your directive above. Using E= would seem to result in the header being set unconditionally, which appears to be what you are seeing. (How LiteSpeed actually interprets E=<var> is unclear – it may just “ignore” it!? In my experience, LiteSpeed seems to just “ignore” directives/syntax that it does not understand – no error is generated, which makes debugging troublesome!)

Aside: On Apache you would also have issues with how environment variables are renamed when the rewriting process starts over – after WordPress rewrites the request to index.php. For example, on Apache you would need to test for REDIRECT_NOINDEXNOFOLLOW, as opposed to NOINDEXNOFOLLOW. However, this does not seem to be the case with LiteSpeed.

RewriteCond %{REQUEST_URI} ^(.*)?faq?(/)$ [NC]
RewriteRule ^faq - [E=NOINDEXNOFOLLOW]

Aside: Your RewriteCond would seem to be unnecessary (and overly complex). And the RewriteCond (condition) and RewriteRule directives conflict to some extent… You have made the RewriteCond case-insensitive, however, the RewriteRule is not, so whilst the condition would match FaQ, the RewriteRule only matches faq (all lowercase). The RewriteCond matches fa (since the trailing q is optional), however, the RewriteRule matches only faq (since the trailing q is not optional here).

So, the above could be written as a single directive:

RewriteRule ^faq/$ - [E=NOINDEXNOFOLLOW]

NB: Note that when setting an environment variable with mod_rewrite, without a stated value, it is set to an empty string. (This is different to mod_env and mod_setenvif that would set the variable to 1.) As it happens, this does not cause a problem with the Header directive.

However, if you are simply setting an env var based on the requested URL then I would use SetEnvIf (mod_setenvif) instead of mod_rewrite. For example:

SetEnvIf Request_URI ^/faq/$ NOINDEXNOFOLLOW

Note that this sets the variable to 1, instead of an empty string.

Summary

You should use the following on LiteSpeed instead:

SetEnvIf Request_URI ^/faq/$ NOINDEXNOFOLLOW
Header set X-Robots-Tag "noindex, nofollow" env=NOINDEXNOFOLLOW

On Apache, you would need to modify the Header directive to test the REDIRECT_NOINDEXNOFOLLOW instead (as mentioned above). For example:

SetEnvIf Request_URI ^/faq/$ NOINDEXNOFOLLOW
Header set X-Robots-Tag "noindex, nofollow" env=REDIRECT_NOINDEXNOFOLLOW

Leave a Comment