Remove Category When Post Transitions from Status Private To Public And has Specific categories already?

Couple of things:

1. You’re not providing a post to check in in_category().
2. You’re calling wp_remove_object_terms() on $postID, which hasn’t been set.

This code should work:

function my_remove_category_on_status_change( $new_status, $old_status, $post ) {
    if ( $old_status == 'private' && $new_status="publish" && in_category( array( '1186', '1208' ), $post ) ) {
        wp_remove_object_terms( $post->ID, 1186, 'category' ); // unset category with id 1186
    }
} 
add_action( 'transition_post_status', 'my_remove_category_on_status_change', 10, 3 );

Note:

1. I’ve passed the $post object from the function parameter to in_category() so that it checks if those categories are on the post that’s changing status.
2. I’ve used $post->ID to give the ID of the post that’s changing status to the wp_remove_object_terms() function.

Also, remove_cat() is a very generic function name, and could easily be a function in WordPress or another plugin, and could result in conflicts. I suggest being more specific and prefixing it with something unique to your project. In my example I used my_.

PS: Hard-coding specific category IDs locks you into those and can make maintenance somewhat difficult. I’d recommend using slugs, which are a bit more predictable, or having a setting somewhere for picking the categories related to this functionality then pulling those values into this function.