WP Plugin: How to avoid db access for each comment?

Yes, I would suspect that your current solution may cause performance issues, if the number of comments gets high.

To make the calculations a little easier for your site, you could handle them before hand based on the post ID. I.e

1. get the post ID
2. query all of the post’s comments’ IDs (WP_Comment_Query or custom $wpdb query)
3. get the votes data from the custom table based on the comment IDs, i.e. WHERE IN query
4. calculate the votes and store the result in a transient, e.g. name
   “comment_vote_counts_{$post_id}” with array value [$comment_id =>
   $vote_count, …], with suitable expiration time

Then either modify your comments loop so that you can feed into it the transient value after reading it once or push the transient value into the WP object cache with wp_cache_set() and read the vote counts with wp_cache_get() in your callbacks.

Instead of using the WP object cache, you could also “cache” the data with a custom object and use it to provide the votes data via a filter. Something along these lines for example,

// provider class
class MyPostCommentsVotesCounts
{
    protected array $votes;

    public function __construct(array $votes)
    {
        $this->votes = $votes;
    }

    public function provide_votes(int $value, int $comment_id): int
    {
        return $this->votes[$comment_id] ?? 0;
    }
}

// get the data or query and calculate it
function wpse_418774_get_comment_vote_counts_for_post(int $post_id): array {
    $counts = get_transient( "comment_vote_counts_{$post_id}" );
    if ($counts && is_array($counts)) {
        return $counts;
    }

    // query and calculate vote counts...
    $counts = [
        // $comment_id => $vote_count,
        // 13 => 100,
        // 22 => 150,
        // 32 => 50,
    ];

    // see https://codex.wordpress.org/Easier_Expression_of_Time_Constants
    set_transient( "comment_vote_counts_{$post_id}", $counts, 5 * MINUTE_IN_SECONDS );

    return $counts;
}

// setup the data provider
function wpse_418774_provide_post_comments_vote_counts(int $post_id): void {
    $votes = new MyPostCommentsVotesCounts(
        wpse_418774_get_comment_vote_counts_for_post($post_id)
    );

    add_filter( 'wpse_418774_comment_vote_counts', [$votes, 'provide_votes'], 10, 2 );
}

// retrieve the votes data by applying a filter
function comment_vote_show_votes(): string {
    $sum_votes = apply_filters( 'wpse_418774_comment_vote_counts', 0, get_comment_ID() );

    return '<div class="comment-vote-votes">' . $sum_votes . '</div>';
}

// setup the provider at some suitable point of the WordPress lifecycle, e.g.
add_action( 'template_redirect', 'wpse_418774_set_up_post_comment_vote_counts' );
function wpse_418774_set_up_post_comment_vote_counts(): void {
    if (! is_single()) {
        return;
    }

    wpse_418774_provide_post_comments_vote_counts(get_the_ID());
}