Can we completely remove the WordPress Sitemaps (WordPress 5.5)?

Disabling the sitemap is easy, just add this line to your functions.php:

add_filter( 'wp_sitemaps_enabled', '__return_false' );

Removing specific posts goes like this:

add_filter(
    'wp_sitemaps_posts_query_args',
    function( $args, $post_type ) {
        if ( 'post' !== $post_type ) {
            return $args;
        }
 
        $args['post__not_in'] = isset( $args['post__not_in'] ) ? $args['post__not_in'] : array();
        $args['post__not_in'][] = 123; // 123 is the ID of the post to exclude.
        return $args;
    },
    10,
    2
);

There a quite a lot of ways to customize the sitemap if you would rather use the WP native way than a plugin.