Either do not use any linebreaks in the LaTeX code, or (what should be preferred) add the following to your functions.php file:
remove_filter('HOPEFULLY-THERE-IS-A-FITTING-FILTER-IN-THE-SHORTCODE-FUNCTION', 'wpautop');
Iff there is such a filter, though.