How to disable block directory?

Yes, it is possible. Excerpt from Editor Hooks → Block Directory in the block editor handbook:

The Block Directory enables installing new block plugins from
WordPress.org. It can
be disabled by removing the actions that enqueue it. In WordPress
core, the function is wp_enqueue_editor_block_directory_assets. To
remove the feature, use
remove_action,
like this:

<?php
// my-plugin.php

remove_action( 'enqueue_block_editor_assets', 'wp_enqueue_editor_block_directory_assets' );

So just use the above code to globally disable the block directory.

• You can also do something like so, which disables the block directory based on the current user’s role:

  add_action( 'init', 'my_disable_block_directory_plugin' );
  function my_disable_block_directory_plugin() {
      // Define a list of allowed user roles.
      $allowed_roles = array( 'administrator' );
  
      // Check whether the current user has any of the above allowed user roles.
      $roles_filtered = array_intersect(
          $allowed_roles,
          (array) wp_get_current_user()->roles
      );
  
      // If the user does not have any of that roles, disable the Block Directory.
      if ( empty( $roles_filtered ) ) {
          remove_action(
              'enqueue_block_editor_assets',
              'wp_enqueue_editor_block_directory_assets'
          );
      }
  }
  

And just so that you know, it can also be disabled via JavaScript:

// This way, you can register it back later, if necessary.
wp.plugins.unregisterPlugin( 'block-directory' );

• Example as a full script:

  wp.domReady( () => {
      if ( wp.plugins.getPlugin( 'block-directory' ) ) {
          wp.plugins.unregisterPlugin( 'block-directory' );
      }
  } );