How to limit comment author to one comment per post

This would depend on how your theme handles saving comments. For instance if it submits the form via ajax it will be expecting certain returns.

The way I would hook into this would probably be through the pre_comment_approved filter. Checking the received email against any previously submitted posts then return unapproved if it fails.

function pre_comment_approved_handler( $approved, $commentdata ) {

    $comments =  get_comments( array(
        'post_id' => $commentdata['comment_post_ID'],
        'author_email' => $commentdata['comment_author_email']
    ) );

    if( !empty($comments) )
        $approved = 0;

    return $approved;
}
add_filter( 'pre_comment_approved' , 'preprocess_comment_handler', 10, 2 );

Untested but something like that should work to straight out block it. Keep in mind email is usually an optional field.

Adding validation feedback to the user is going to be a problem when comments are involved as hooks and filter for that don’t really exist. The best way would be to submit the form via ajax and check the email at that point and if it snot made a post then you can create the comment manually. That way you’ll have full control over the user experience.