How to limit characters of the post title?

You could do this :

add_action('publish_post', 'wpse_107434_title_max_char');
function wpse_107434_title_max_char() {
  global $post;
  $title = $post->post_title;
  if (strlen($title) >= 100 )
    wp_die( "the title must be 100 characters at most" );
  }

You can replace strlen() with str_word_count() if you want to set a word limit instead.

EDIT: ok with new details you added it seems you could add some jQuery to do the same (strlen)