Here is a nice tutorial I’ve read http://www.garyc40.com/2010/03/how-to-make-shortcodes-user-friendly/
He as a great example and you cal also download the source files and get a better understanding
That’s how I added a custom button to the tinymce editor in my plugin simple TOC which you can also download , look at the code and get an even better understanding.