Is there a (preferable built-in) way to check what custom queries are used in a theme?

I don’t really understand what you’re trying to do, so here goes nothing:

There are a lot of hooks within WP_Query. Most of them pass the current instance as one of the arguments. So, here’s how you could check for a “custom query”:

function check_wp_query( $posts, $wp_query ) {
  if ( $wp_query->get('my_arg') ) {
    // it's a custom query
  }

  return $posts;
}
add_filter( 'the_posts', 'check_wp_query', 10, 2 );

The trick is that each custom query has to be identifiable by at least one of it’s query vars.