How to set all external links to a certain domain to “nofollow”?

Assuming you are targetting links in post content, I’d say something like that :

add_filter( 'the_content', 'wpse_254317_nofollow' );
/**
 * Add rel nofollow according to a predefined domain
 * to links in content
 * @param $content
 * @link http://stackoverflow.com/a/4809181/1930236
 * @return mixed
 */
function wpse_254317_nofollow( $content ) {

    $my_folder = "https://google.com";
    preg_match_all( '~<a.*>~isU', $content, $matches );

    for ( $i = 0; $i <= sizeof( $matches[0] ); $i ++ ) {
        if ( isset( $matches[0][ $i ] ) && ! preg_match( '~nofollow~is', $matches[0][ $i ] )
             && ( preg_match( '~' . preg_quote( $my_folder ) . '~', $matches[0][ $i ] )
                  || ! preg_match( '~' . get_bloginfo( 'url' ) . '~', $matches[0][ $i ] ) )
        ) {
            $result = trim( $matches[0][ $i ], ">" );
            $result .= ' rel="nofollow">';
            $content = str_replace( $matches[0][ $i ], $result, $content );
        }
    }

    return $content;
}

The code matches all the links in post content and add the wanted rel for the domain set as $my_folder which is “https://google.com” in my example.