About redundant code when I use shortcode to output the HTML from a variable

This is because the content of a post – and therefore your shortcode – is run through the wpautop() function which generates the <p> and <br /> tags in order to space out the post properly.

Shortcodes are run at priority 11, which is after wpautop() on priority 10. We can change the piority that this shortcode is run before wpautop():

// This will do nothing but will allow the shortcode to be stripped
add_shortcode( 'foobar', 'shortcode_foobar' );

// Actual processing of the shortcode happens here
function foobar_run_shortcode( $content ) {
    global $shortcode_tags;

    // Backup current registered shortcodes and clear them all out
    $orig_shortcode_tags = $shortcode_tags;
    remove_all_shortcodes();

    add_shortcode( 'foobar', 'shortcode_foobar' );

    // Do the shortcode (only the one above is registered)
    $content = do_shortcode( $content );

    // Put the original shortcodes back
    $shortcode_tags = $orig_shortcode_tags;

    return $content;
}

add_filter( 'the_content', 'foobar_run_shortcode', 7 );

Code thanks to @Viper007Bond ^[source]. Remember to replace the function and filter names with ones relating to your shortcode!