This isn’t the way shortcode works. When you use shortcode it isn’t processed in the editor, it is processed when someone views the page.
It sounds like what you really want is to add a custom button to TinyMCE and from there the user can select a table template (or whatever) that gets insert into the editor.
Here are some good resources for this: