Changing author slug for a custom role without using plugin

You can use add_permastruct() (with ep_mask set to EP_AUTHORS) to add the proper rewrite rules, and the author_link filter to set the proper author URL:

add_action( 'init', function(){
    add_permastruct( '%author_trip_vendor%', 'operator/%author%', [
        'ep_mask' => EP_AUTHORS,
    ] );
} );

add_filter( 'author_link', function( $link, $author_id, $author_nicename ){
    if ( user_can( $author_id, 'trip_vendor' ) ) {
        $link = '/operator/' . $author_nicename;
        $link = home_url( user_trailingslashit( $link ) );
    }
    return $link;
}, 10, 3 );

Don’t forget to flush the rewrite rules — just visit the permalink settings page.

UPDATE

non-vendors profile are loaded on URL example.com/operator/admin and
also on example.com/author/admin

You can fix it via one of these options:

1. Send a “404” header (“Page not found” error)

   add_action( 'parse_request', function( $wp ){
       if ( preg_match( '#^(author|operator)/([^/]+)#', $wp->request, $matches ) ) {
           $user = get_user_by( 'login', $matches[2] );
           if (
               ( 'author' === $matches[1] && $user && user_can( $user, 'trip_vendor' ) ) ||
               ( 'operator' === $matches[1] && $user && ! user_can( $user, 'trip_vendor' ) )
           ) {
               $wp->query_vars = ['error' => 404];
           }
       }
   } );
   

2. Redirect the user to the proper “operator” URL

   add_action( 'parse_request', function( $wp ){
       if ( preg_match( '#^(author|operator)/([^/]+)#', $wp->request, $matches ) ) {
           $user = get_user_by( 'login', $matches[2] );
           if (
               ( 'author' === $matches[1] && $user && user_can( $user, 'trip_vendor' ) ) ||
               ( 'operator' === $matches[1] && $user && ! user_can( $user, 'trip_vendor' ) )
           ) {
               $base = ( 'author' === $matches[1] ) ? 'operator' : 'author';
               wp_redirect( home_url( "https://wordpress.stackexchange.com/" . $base . "https://wordpress.stackexchange.com/" . $matches[2] ) );
               exit;
           }
       }
   } );