Password protected post or page – error message by wrong password?

not really, but can try something like:

add_action('wp', 'check_post_pass');

function check_post_pass(){

  if(!is_single() || !post_password_required()) return;

  global $post;
  if(isset($_COOKIE['wp-postpass_'.COOKIEHASH])
      && $_COOKIE['wp-postpass_'.COOKIEHASH] !== $post->post_password){

    define('INVALID_POST_PASS', true);

    // tell the browser to remove the cookie so the message doesn't show up every time
    setcookie('wp-postpass_'.COOKIEHASH, NULL, -1, COOKIEPATH);
  }

}   

in your template:

if(defined('INVALID_POST_PASS')) _e('The password you entered is funky');

But a much better idea would be to create your own the_content()-like function and password form + check function, without cookies…