Which escaping function should be use on register_post_type label?

None. Escaping should happen late, on output. This is just registering the strings for later use, so it’s too early to escape. All the places in WordPress where the post type label is used automatically, WordPress should already be escaping it.

If you’re outputting any of the labels yourself, you would probably use esc_html():

$post_type_object = get_post_type_object( 'book' );
$post_type_labels = get_post_type_labels( $post_type_object );

echo esc_html( $post_type_labels->singular_name );