You need to look at this from a couple of perspectives and then you have to weigh the odds against each other
In general, shortcodes will be slower (we are talking milliseconds here) as they need to be parsed and processed by the shortcode handler.
USAGE: [my_shortcode] in post editor
Shortcodes in general are quite useful and usally outweigh the fact that they tend to be tad slower than just hardcoding a function and calling it in the desired template. Shortcodes are portable (that is why they should be in a plugin) which makes them excellent for usage between themes. More so, you do not need to alter any template in order to do something special in a template, you can simply drop the shortcode into the post editor’s content meta box and your done.
For convenience and ease of use, shortcodes are definitely the way to go here, and all the positives here outweigh the loss in a mere few milliseconds of time due to not using a spaghetti callback function on its own
USAGE: do_shortcode() hardcoded in a template
This is where I would draw the line, if you need to call a shortcode via do_shortcode(), then you are doing it wrong. As I said, shortcodes are a tad slower, so here it would be way better to just write a simple function, drop it into a plugin (which is recommended for portability) or functions.php, and then call the function into your template.
Either way, you will have to alter the template to either call your shortcode or the function or even class which you created. This is why, IMHO, you should not be using shortcodes, but rather a simple function or a class which is a bit faster.
CONCLUSION
Should you stay away or limit the use of shortcodes? The answer is simple, it all depends on usage as described above. Just a final note, if you are going to call multiple shortcodes within a page, try to limit those by combining features you need into one solid, clever shortcode which can do everything you want.
Before I lay my pen down, I want to leave you with this, download and install the Query Monitor plugin. I personally feel that every developer should have this plugin installed on his/hers local test enviroment. This plugin will tell you excatly how many queries are being made, time take to make those queries, and it will also tell you how much time it took to load a specific page.