How can I remove “”

The default Shortcode will add that markup when necessary and only if the active WordPress theme didn’t enable HTML5 support for the gallery Shortcode’s markup/output. Or, in rare cases, a plugin or a custom code may have disabled the support.

So an easy way to get rid of that <br> tags, is by enabling HTML5 support for the Shortcode, like so: (add to the theme’s functions.php file)

add_theme_support( 'html5', 'gallery' );

Or without enabling the HTML5 support, you can also use CSS to visually hide those <br> tags:

.gallery > br {
    display: none;
}

For my custom output I would prefer something like
this.

In that case, you can copy the code in gallery_shortcode(), which is the default callback function for the gallery Shortcode, and then just modify the gallery markup to your liking. Here’s an example which I tried and tested working on WordPress 4.9.8: (full code here)

// See gallery_shortcode() for reference.
add_filter( 'post_gallery', 'my_post_gallery', 10, 3 );
function my_post_gallery( $output, $attr, $instance ) {
    $post = get_post();

    // Change #1: I commented out this part; use the passed $instance (see above).
    /*static $instance = 0;
    $instance++;*/

    ...

    // Change #2: I commented out this part; otherwise, the page would stop working.
    /*$output = apply_filters( 'post_gallery', '', $attr, $instance );
    if ( $output != '' ) {
        return $output;
    }*/

    ...

    $i = 0;
    foreach ( $attachments as $id => $attachment ) {

        ...
        // Change #3: I commented out this code.
        /*if ( ! $html5 && $columns > 0 && ++$i % $columns == 0 ) {
            $output .= '<br style="clear: both" />';
        }*/
    }

    // Change #4: I commented out this code.
    /*if ( ! $html5 && $columns > 0 && $i % $columns !== 0 ) {
        $output .= "
            <br style="clear: both" />";
    }*/

    // Change #5: Here's the only one `br` tag we need.
    $output .= "
            <br style="clear: both" />
        </div>\n";

    return $output;
}