add_rewrite_rule() driving me crazy, rewrite not working when analyzer says it should

Your external rule doesn’t work because you’re using a format that only works for internal rules- $matches[1] is essentially nonsense in the context of an .htaccess file. Using the $matches array will only work when rules are parsed by PHP, external rules need to use the standard $1 instead of $matches[1]. Any rule that doesn’t point to index.php is interpreted as an external rule and gets added to .htaccess instead of the internal rules array.

That said, I suggest changing things a bit and adding a rewrite endpoint instead of a rule. Your URLs would be:

http://example.com/category/postname/ajax/

You could then check if get_query_var('ajax') is set in your templates.