The get_the_author_meta function gets the Post’s author ID, not the comment-author’s information.
The get_avatar() function is perfectly capable of accepting the comment object whole and working it out from that all by itself. So just call it like so:
echo get_avatar( $comment, 32 );