Before loading plugins WP retrieves the list of active and valid plugins with the function wp_get_active_and_valid_plugins. The array of plugins is retrieved in this line:
$active_plugins = (array) get_option( 'active_plugins', array() );
As you can see, this involves a call to get_option, which has a filter to manipulate the result. So if you could build a filter like this:
add_filter ('option_active_plugins','wpse406546_filter_plugins') {
// remove the plugin from the array
}
Note that you cannot do something depending on user at this stage, because the current user has not been authenticated yet (see hook order). Also, you must place this filter in a must use plugin, because it must be loaded before WP starts loading the regular plugins.
Further on in the loading process, at the init hook, the user is known and you can build an action there to load the plugin depending on current user.
Beware that there may be unexpected complications if you use this on plugins that you did not develop yourself, because late loading may make the plugin miss a hook. For instance, if you load a plugin at init and the plugin uses the after_setup_theme hook, that will be too late.