My understanding is that you want to set number of posts limit on your categories, so that each category will have specified number of posts. After a very long time I am trying to answer a question on WordPress SE, so I hope I will make sense.

Explanation

Categories does not have meta data, so you are going to need your own custom table to store category limits. Reading and writing to the custom table should be abstracted in your custom meta functions. For example get_post_count_limit. Once the get and set functions are ready, then you can then hook up to your desired WordPress actions (as you mentioned publish post, so in the code below I have used the publish_post action).

In the function hooked to publish_post action, you simple loop over all the categories assigned to the post, check their limit using the custom function and then check the category_count property. If category_count exceeds the limit, simply fetch the oldest posts and remove category from them.

Code

<?php
/**
 * Plugin Name: Category Post Count Limit
 * Description: Allows to set limit on number of posts in a category
 * Author: Hameedullah Khan
 **/



add_action( 'publish_post', 'cpcl_check_post_limit' );
function cpcl_check_post_limit( $post_id ) {
    $cat_ids = wp_get_post_categories( $post_id );
    foreach( $cat_ids as $cat_id ) {
        $cat = get_category( $cat_id );
        $post_count_limit = get_post_count_limit( $cat_id );
        if ( $post_count_limit < 1 ) { // limit lower then 1 means unlimited.
            continue;
        }
        $category_count = $cat->category_count;
        if ( $category_count > $post_count_limit ) {
            $numposts = $category_count - $post_count_limit;
            $query = new WP_Query( array( 'category__in' => array( $cat_id ), 'orderby' => 'date', 'posts_per_page' => $numposts, 'order' => 'ASC' ) );
            while ($query->have_posts() ) {
                $query->next_post();
                $cp_cat_ids = wp_get_post_categories( $query->post->ID );
                unset( $cp_cat_ids[array_search( $cat_id, $cp_cat_ids )] );
                $cp_cat_ids = array_values( $cp_cat_ids );
                wp_set_post_categories( $query->post->ID, $cp_cat_ids );

            }
            wp_reset_postdata();
        }
    }
}

add_action( 'category_add_form_fields', 'cpcl_field_add', 10, 1);
function cpcl_field_add( $cat ) {
?>
    <div class="form-field">
        <label for="post-count">Post Count Limit</label><input type="text" name="post-count" type="text" /></label>
    </div>
<?php
}

add_action( 'category_edit_form_fields', 'cpcl_field_edit', 10, 1 );
function cpcl_field_edit( $cat ) {
    $post_count = get_post_count_limit( $cat->term_id );
?>
    <tr class="form-field">
        <th scope="row" valign="top">
            <label for="post-count">Post Count Limit</label>
        </th>
        <td>
            <input type="text" name="post-count" type="text" value="<?php echo $post_count; ?>" /></label>
        </td>
    </tr>
<?php
}


add_action( 'created_category', 'cpcl_save_post_count_limit', 10, 1 );
add_action( 'edited_category', 'cpcl_save_post_count_limit', 10, 1 );
function cpcl_save_post_count_limit( $cat_id ) {
    if (current_user_can( 'manage_categories' ) && isset( $_POST['post-count'] ) ) {
        $post_count_limit = ( int ) $_POST['post-count'];
        update_post_count_limit( $cat_id, $post_count_limit );
    }

}

function get_post_count_limit( $cat_id, $return_null = false ) {
    global $wpdb;

    $table_name = $wpdb->prefix . 'cat_post_count_limit';

    $cpcl_meta = $wpdb->get_row( "SELECT * FROM $table_name WHERE cat_id = $cat_id" );

    if ( ! $cpcl_meta ) {
        if ( $return_null && $cpcl_meta == null ) {
            return null;
        } else {
            return 0;
        }
    } else {
        return $cpcl_meta->post_count;
    }
}
function update_post_count_limit( $cat_id, $post_count_limit ) {
    global $wpdb;

    $table_name = $wpdb->prefix . 'cat_post_count_limit';

    if ( get_post_count_limit( $cat_id, true) == null )  {
        $wpdb->insert( 
            $table_name, 
            array(
                'cat_id' => $cat_id,
                'post_count' => $post_count_limit
            ),
            array(
                '%d',
                '%d'
            )
        );
        return $wpdb->insert_id;
    } else {

        return $wpdb->update(
            $table_name,
            array(
                'post_count' => $post_count_limit
            ),
            array(
                'cat_id' => $cat_id
            ),
            array( '%d' ),
            array( '%d' )
        );
    }
}

function cpcl_initialize_categories() {
    global $wpdb;

    $table_name = $wpdb->prefix . 'cat_post_count_limit';

    $sql = "CREATE TABLE IF NOT EXISTS `$table_name` (
        `cpcl_id` int(11) NOT NULL AUTO_INCREMENT,
        `cat_id` int(11) NOT NULL,
        `post_count` int(11) NOT NULL,
        PRIMARY KEY (`cpcl_id`)
    )";
    require_once( ABSPATH . 'wp-admin/includes/upgrade.php' );
    dbDelta( $sql );
}
register_activation_hook( __FILE__, 'cpcl_initialize_categories' );
?>

Note: Use the above code as plugin, so the custom table can be create on plugin creation. If you don’t want to use it as plugin you will have to create the custom table manually.

Really looking forward for suggestions on improving my answer and/or code.