Your problem is that you installed it in a subdirectory of WordPress. Both WordPress and Gallery 3 will use virtual URLs in their domain, and with them both acting on the same folders, neither system knows what to do.
So you have http://blog.url/ and http://blog.url/gallery3/ which is also http://g3.blog.url/.
The subdomain will point to that folder without any problem … http://g3.blog.url/ will redirect to http://blog.url/gallery3/ … but http://g3.blog.url/something will redirect to http://blog.url/gallery3/something.
Logically, you’ll think this will work, but both Gallery 3 and WordPress will try to eat this URL to pass it to the appropriate page. WordPress will send http://g3.blog.url/login/ajax to http://blog.url/index.php/login/ajax because that’s where it thinks you’re trying to go.
My recommendation would be to not use a subfolder of the WordPress directory for Gallery 3. Put it in a separate folder altogether and point your subdomain there. Then tell WordPress to reference the subdomain rather than the subfolder. You won’t run into .htaccess collisions that way.