Limit the number of plugins that can be installed in a WordPress installation

The most performant way to do this is probably by hooking into map_meta_cap to conditionally disallow the install_plugins capability. There are no database calls involved with that.

Here’s an (untested) example:

add_filter( 'map_meta_cap', 'wpse291964_limit_plugins', 10, 2 );

function wpse291964_limit_plugins( $caps, $cap ) {
  if ( 'install_plugins' === $cap ) {
    $plugins = get_plugins();

    // Only allow 10 plugins to be installed.
    if ( count( $plugins ) > 10 ) {
      $caps[] = 'do_not_allow';
    }
  }

  return $caps;
}

I recommend you to watch this video by John Blackbourn to learn more about map_meta_cap and friends.