Remove ‘read more’ link when post is shorter than excerpt

Are you using the_excerpt() or $post->post_excerpt? Use $post->post_excerpt and count your words then if the limit exceeds show the read more otherwise don’t show.