How do I turn off self-closing tags for markup in WordPress (for HTML5, or HTML4, for example)?

Line breaks are added by wpautop(), not wptexturize(). wpautop() is also the function that automatically adds paragraph tags.

You’re better off fixing the <br />‘s than you are replacing the filter. Since wpautop() runs at priority 10, you can just hook in after that and fix it.

add_filter( 'the_content', 'html5_line_breaks', 25 );

function html5_line_breaks( $content ) {
    return str_replace( '<br />', '<br>', $content );
}

Edit after OP update:

WordPress functions are designed to output XHTML. In order to get rid of those trailing slashes site-wide, you’re going to have to use an output buffer. You could use a filter similar to the one above to replace slashes in the post contents, but that wouldn’t catch your head, sidebar, etc.

It’s a bit ugly and might have a small impact on performance, but here you go (drop this in a plugin or your theme’s functions.php file):

if ( !is_admin() && ( ! defined('DOING_AJAX') || ( defined('DOING_AJAX') && ! DOING_AJAX ) ) ) {
    ob_start( 'html5_slash_fixer' );
    add_action( 'shutdown', 'html5_slash_fixer_flush' );
}

function html5_slash_fixer( $buffer ) {
    return str_replace( ' />', '>', $buffer );
}

function html5_slash_fixer_flush() {
    ob_end_flush();
}

That code says if you’re not in the administration area and not doing an AJAX request handling, then start buffering the output through a filter and then using the WordPress shutdown hook, output that buffer.