How to limit the number of forgot password reset attempts in WordPress?

You need put the attempts on user meta, then do check every time user hit reset passwords.

add_action( 'password_reset', 'my_password_reset', 10, 2 );
function my_password_reset( $user, $new_pass ) {
    $limit=5;// Set the limit here
    $attempts=(int) get_user_meta($user->ID,"reset_attempts",true);
    if($attempts>$limit){
       //Do something in here, example redirect to warning page.
       wp_redirect( "/warning" );
       exit;
    } 
    update_user_meta($user->ID,"reset_attempts",$attempts++);
}