robots.txt - Read For Learn

Question about robots.txt file on WordPress site

Here is some info about robots.txt. It looks like your example wouldn’t work correctly with the * but, it looks to me like you’d be ok if you just removed the *. You can always check the robots.txt at with a validator like this one

Why are some user agents still getting a meta robots tag with noindex from my site?

I was editing stuff in default-filters.php and then cleared my wp cache and it finally went away. Then I undid my change to default-filters.php and it stayed away. So I’m inclined to think, this was a caching issue?

Robots.txt file not updating

How to noindex, follow a specific category wordpress?

There are several robots.txt generators online. Here is one of my favorites. After you’ve generated your file, I highly recommend testing it with Google’s Webmaster Tools tester. I’ve done this for dozens of sites, and never have a problem. Just remember the following: Most WordPress sites treat categories as if they’re folders/directories. So, to make … Read more