Filter “wp_mail” – How to distinguish between a WordPress system email and a plugin email?

I agree with @birgire that #53829-core would be the ideal way to do this. If you need a hack that works in the meantime, though, then you can examine the filenames in the call stack.

add_filter( 'wp_mail', 'modify_system_emails' );

function modify_system_emails( array $attributes ) : array {
    $wp_mail_caller = get_wp_mail_caller();

    if ( is_core_file( $wp_mail_caller['file'] ) ) {
        $attributes['subject'] .= ' -- (system)';
    }

    return $attributes;
}

function get_wp_mail_caller() : array {
    $backtrace = debug_backtrace( DEBUG_BACKTRACE_IGNORE_ARGS );

    foreach ( $backtrace as $call ) {
        if ( 'wp_mail' === $call['function'] ) {
            return $call;
        }
    }

    throw new Exception( "`wp_mail()` wasn't found in the backtrace." );
}

function is_core_file( string $filename ) : bool {
    $relative_path = str_replace( ABSPATH, '', $filename );
    $parts         = explode( "https://wordpress.stackexchange.com/", $relative_path );

    if ( in_array( $parts[0], [ 'wp-admin', 'wp-includes' ], true ) ) {
        return true;
    } elseif ( preg_match( '#^wp-.*\.php$#', $parts[0] ) ) {
        return true;
    }

    return false;
}

You can test that with some code like:

add_action( 'init', function() {
    wp_mail( '[email protected]', 'not a system email', 'nosirree' ); // This won't modify the subject.
    wp_site_admin_email_change_notification( '[email protected]', '[email protected]', 'foo' ); // This will modify the subject.
} );