How to select category on if function

According to the documentation, transition_post_status action receives only 3 parameters ($old_status, $new_status, and $post).

There is no $category_id parameter, but as post object is available for you, you are able to get post category using this post object.

In your case, for the if statement, it’s better to use has_category() function where you can pass category ID and post object.

has_category()
Checks if the current post has any of given category.

So you filter code should look like this:

function telegram_send_message( $new_status, $old_status, $post) {

    //$category_id=='1766' changed to has_category(1766, $post)
    if( $new_status == 'publish' &&  $old_status != 'publish' && $post->post_type == 'post'&& has_category(1766, $post) {
        $apiToken = "TOKEN";
        $data = [
            'chat_id' => '@******',
            'text' => "\nRead more: " . get_permalink($post->ID)
        ];
        $response = file_get_contents("https://api.telegram.org/bot******/sendMessage?" . http_build_query($data) );
    }

}

add_action( 'transition_post_status', 'telegram_send_message', 10, 3 );