How to add a nested shortcode into editor?

You can do it developing a TinyMCE plugin and load it using mce_external_plugins hook. The key is in the javascript file of the TinyMCE plugin which will be responsible to pass the nested shortcodes into the editor textarea. You can take a look to this article (about the middle they shows an example of how to add a shortcode button in the editor) or this one.