How to make comments private for commentor and post author

You can use the pre_get_comments filter to modify the parameters of the comment query before it fetches the comments. Specifically the author_in parameter.

I tried to write an example, though I haven’t tested it, but it would be similar to this:

add_action( 'pre_get_comments', 'author_and_self_comment_filter' );

function author_and_self_filter( \WP_Comment_Query $query ) : void {
    // We need to do some checks first and return early if
    // this filter doesn't apply, e.g. if you're the post
    // author etc

    // don't break the admin UI
    if ( is_admin() ) {
        return;
    }

    // only filter if we're grabbing the comments for a post
    if ( $query->query_vars['post_id'] === 0 ) {
        return;
    }

    // only logged in users should see comments
    if ( ! is_user_logged_in() ) {
        // we need to return 0 results so I'm asking for a
        // comment type that doesn't exist so there are no
        //results, there's probably a better way to do this:
        $query->query_vars['type'] = 'banana';
        return;
    }

    // this is already filtered to specific people! Skip!
    if ( ! empty( $query->query_vars['author__in'] ) ) {
        return;
    }

    // Admins + super admins see everything!
    if ( current_user_can( 'manage_options' ) || is_super_admin() ) {
        return;
    }

    // get my ID
    $my_user_id = get_current_user_id();

    // get the post author ID
    $post_id = $query->query_vars['post_id'];
    $p = get_post( $post_id );
    $post_author_id = $p->post_author;

    // If I'm the post author then I can see everything
    if ( $my_user_id === $post_author_id ) {
        return;
    }

    // now we need to set the `author_in` to an array with 2
    // values, the authors user ID and the current users ID
    $authors = [];
    $authors[] = $my_user_id; // My user ID
    $authors[] = $post_author_id; // Author ID
    $query->query_vars['author_in'] = $authors;
}

There’s a series of checks to figure out if we need to do this or not, and then at the end we restrict comments to just the author and current user.

Note that comment counters will never be accurate if you do this and may be highly unreliable, there is no fix for this without undoing the restriction so all comments are visible to all users.