What’s the earliest point I can get the queried object ID?

A more compact and WordPress-ish way, using url_to_postid():

function wpmp_switcher_exclusions_init() {
  $pid = url_to_postid( home_url( add_query_arg( array() ) ) );
  if (
    (int) $pid > 0 &&
    get_post_meta( $pid, '_wp_page_template', TRUE ) === 'my-responsive-template.php'
   ) {
    remove_action('init', 'wpmp_switcher_init');
    remove_action('admin_menu', 'wpmp_switcher_admin_menu');
    remove_action('wp_footer', 'wpmp_switcher_wp_footer');
    remove_filter('stylesheet', 'wpmp_switcher_stylesheet');
    remove_filter('template', 'wpmp_switcher_template');
    remove_filter('option_home', 'wpmp_switcher_option_home_siteurl');
    remove_filter('option_siteurl', 'wpmp_switcher_option_home_siteurl');
  }
}

add_action( 'setup_theme', 'wpmp_switcher_exclusions_init' );