“pre_get_posts” firing on every query

Basically what you are looking for is the global $wp_the_query variable which is set to the value of the main query. It may not be a perfect fit for 100% of cases but will probably work fine in 99% of cases:

add_action( 'pre_get_posts', 'custom_post_count' );
function custom_post_count( $query ){
  global $wp_the_query;
  if ( $wp_the_query === $query ) {
    $query->set( 'posts_per_page', 5 );
  }
  return $query;
};