1) Don’t alter or edit core WordPress files or folders.
2) There is no function in WordPress core that dynamically generates a robots.txt file. The theme you are using is trying to create the robots.txt file with that function in functions.php and must be creating it with the wrong character encoding, and that is causing the 500 Server Error.
3) Disable that function – ask the theme developer how – and make your own robots.txt file with an FTP client with the character encoding of Unicode-no BOM.
The functions do_robotstxt runs in the do_robots function before it prints out the Disallow lists for the robots.txt file. http://adambrown.info/p/wp_hooks/hook/do_robotstxt
It’s the function WordPress uses to block indexing of sites. The theme developer has “hijacked” the function incorrectly.
A sample robots.txt file for WordPress sites from http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress
Sitemap: http://www.example.com/sitemap.xml
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
# digg mirror
User-agent: duggmirror
Disallow: /
# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/