First of all, make sure you try things out in a development environment on your own PC. This way you can try things out without breaking your live site. The XAMPP package contains everything you need to run it on your own Windows, Mac, Linux or Solaris machine.
Then, enable debugging mode by setting WP_DEBUG to true in your wp-config.php. You need this to display errors, otherwise you get the infamous “white screen of death” (basically WordPress ran into an error, but for security reasons doesn’t show everyone what it is, leaving you to wonder what went wrong).
With this basic setup you can start playing with functions.php. Make sure you don’t add an ending ?> tag: PHP doesn’t need it, and you might inadvertently add a blank line after it, which will start the output and prevent sending HTTP headers later (for redirection or cookies, like on login). So your file starts with <?php, and then contains only PHP code.
It’s also a good idea to disable all other plugins and use a basic theme. This way, you can be sure you are the only using the different hooks. Or if you use plugins, enable them one by one and test your site at each step, so you know what plugin causes an error. (Enabling WP_DEBUG might show you many notices indicating sloppy coding style, even in “official” plugins – these do not always lead to errors, but they don’t improve your confidence in the code.)
After this you should follow the specific error messages you get. I also use the Xdebug PHP extension to get easy-readable traces of error messages, and I recommend you do it too.