How to disable wp_mail for all users except for admins?

You could use the email address to get the user object, which will include the user ID. You can use the user ID to check the user’s capabilities to see if they have a capability assigned to the admin user role, and if not, unset the address:

add_filter('wp_mail','disabling_emails');
function disabling_emails( $args ){
    $user = get_user_by( 'email', $args['to'] );
    if ( ! user_can( $user->ID, 'create_users' ) ) {
        unset ( $args['to'] );
    }
    return $args;
}

Some notes about what is being done here:

• Note that I used user_can() to check a capability. People tend to use current_user_can() and user_can() to check if a user has a role. This is an improper use of the function. You can do it, and it will generally work, but why do it wrong when you can do it right? Pick a capability that users you intend to send email to will have that other users will not have. In a default WP setup, only admins will have create_users capability, so this is a good one to check against.
• I would be judicious in using this. Depending on how users interact with the site (if at all), what if a user needs to reset a forgotten password? A non-admin user will not be able to receive a password reset.