Add popup window to TinyMCE buttons
Here is a nice and a fairly new (a year old) tutorial that i’ve read a few days ago: http://solislab.com/blog/how-to-make-shortcodes-user-friendly/ He has a great example and you can also download the source files and get a better understanding.