If you look in wp-includes/default-filters.php
you’ll see all of the functions each comment is run through before it’s output. I’d guess it’s the last one, wpautop, which adds p tags in place of line breaks:
add_filter( 'comment_text', 'wptexturize' );
add_filter( 'comment_text', 'convert_chars' );
add_filter( 'comment_text', 'make_clickable', 9 );
add_filter( 'comment_text', 'force_balance_tags', 25 );
add_filter( 'comment_text', 'convert_smilies', 20 );
add_filter( 'comment_text', 'wpautop', 30 );
You can remove_filter( 'comment_text', 'wpautop', 30 );
to confirm, but you’ll lose paragraphs entirely.