Filters and Actions
The hooks you can chose to run whatever limits the post content length from, is either save_post, save_post_{post_type} or edit_post (or both) (the following actions are executed inside wp_publish_post()
/** This action is documented in wp-includes/post.php */
do_action( 'edit_post', $post->ID, $post );
do_action( "save_post_{$post->post_type}", $post->ID, $post, true );
do_action( 'save_post', $post->ID, $post, true );
do_action( 'wp_insert_post', $post->ID, $post, true );
or one of the post transition hooks executing inside wp_transition_post_status(). Namely those are:
do_action( 'transition_post_status', $new_status, $old_status, $post );
do_action( "{$old_status}_to_{$new_status}", $post );
do_action( "{$new_status}_{$post->post_type}", $post_id, $post );
To check inside a callback on an filter or action if you are on the correct filter, you can use current_filter() or current_action() (where later is just a wrapper for current_filter()), which returns a string with the name of the filter or action currently running.
Limits
To limit the amount of words, you may want to use
wp_trim_words( $content, 300 );
to limit the content to for e.g. 300 words.
There is more to it when there is already content present. To get around this, you can leverage the plugins in this answer I wrote to limit excerpt length. Just alter it if you want to run it on the content only.
The general idea (in pseudo code) always is:
Explode the string of words in an array on empty space
Count the number of array items
If they exceed $limit, loop through them or slice the array
Return everything below and equal to the threshold
Core API
The reason why you may want to use wp_trim_words() instead of a simple loop is, that it strips all HTML tags from a string (so they don’t add to the count) and that there is a filter running callbacks before the result is returned, allowing for a very fine grained and targetted approach
apply_filters( 'wp_trim_words', $text, $num_words, $more, $original_text );
Putting a plugin together
Just fill in whatever you may want to do. Make sure that you inform the user properly by using something like post_updated_messages to tell the user why you stripped off words, refused to save the content or whatever UX you want deliver
<?php
/** Plugin Name: WPSE (#172544) Limit Content by Word Count */
add_action( 'save_post_test_post_type', function( $id, \WP_Post $post, $updated )
{
# debug:
# var_dump( $post, $_POST );
// Here you can do whatever you want to do to limit the
}, 10, 3 );