shortcodes – style them or change/remove the square brackets?

While I am sure that this is theoritically possible, I am going to say “No, not really”.

If you look at the source, you can see that those brackets are hard-coded into a very complicated regex. I see no filters that will let you change that.

The only reasonable way to approach this that I can think of would be:

1. To remove the default shortcode system using
   remove_filter('do_shortcode','the_content');
2. Add your own shortcode system

This, of course, will break any plugin supplied shortcodes as well as the core ones. So the other option would be to:

1. Add your own “wrapper” that converts whatever pattern you want to
   use into standard shortcodes
2. And run do_shortcode on the result before returning the result

I think you should be able to preserve the functionality of Core and Plugin shortcodes as well as add your own pattern, but the regex you are needing is complex especially given the pattern you require.

Either option is far more code than I have time to address, not to mention the debugging that would be involved.