I would hook that on publish_post, so you aren’t messing with drafts, etc.
And you need to account for multiple categories, and count the number of posts in each, and delete anything over 10.
Perhaps something like this will get you in the right direction. May need some tweaking, mostly untested.
function on_post_publish( $ID, $post ) {
$cat = get_the_category( $ID ); //returns array of categories
foreach ($cat as $c) { //doing the rest for each cat in array
$args = array(
'orderby' => 'post_date', //using the post date to order them
'order' => 'ASC', // putting oldest at first key
'cat' => $c, // only of the current cat
'posts_per_page' => -1, //give us all of them
'fields' => 'ids' //only give us ids, we dont want whole objects
);
$query = new WP_Query( $args );
$count = $query->post_count; //get the WP_Queries post_count object value
if ($count > 10) { // if that value is more than 10
$id_to_delete = $query->posts[0]; //get oldest post from query
wp_delete_post( $id_to_delete ); //delete it
}
}
wp_reset_postdata(); //resetting to main query just in case
}
add_action( 'publish_post', 'on_post_publish', 10, 2 );