Prevent shortcode from being wrapped in tags

Instead of trying to remove the automatically added paragraphs before or after the fact, you can try removing the wpautop filter from the_content within your shortcode (and re-adding it to maintain consistency for other plugins etc.):

add_shortcode( 'test', 'test_sc' );
function test_sc( $atts ){
    remove_filter( 'the_content', 'wpautop' );
    $content = apply_filters( 'the_content', '$\frac{15}{5} = 3$' );
    add_filter( 'the_content', 'wpautop' );
    return $content;
}