The number for the first page is always 1. The number of maximum pages can be set to $max_page by calling get_comment_pages_count():
$max_page = get_comment_pages_count();
Probably this information helps on how to build links on top of that, which depends a bit on the style of your theme. For building links, I could not find any suitable function in the Worpdress API so far, the most promising seems paginate_links() to me.