register_setting() uses Settings API, which is just an API for options.php, is easier to use, more secure and preffered way of storing options since WordPress 2.7.
If you need to store only a few options or doing simple plugin just for yourself – it’s really up to you which one to use 🙂
Here’s the Settings API (well) explained: http://www.chipbennett.net/2011/02/17/incorporating-the-settings-api-in-wordpress-themes/
Basically it makes a few things much easier, you don’t have to render whole forms all over again (great for plugins/themes with many options), you don’t need to care about nonces etc.
So, to sum up, both these functions actually do the same thing 🙂