How to check if a post belongs to a category that has only 1 posts?

Use get_the_category to get the categories belonging to the post. You can then use $category->count to return the amount of posts for specific category attached to the post

Example:

If a post has just one category attached to it, you can do the following

$category = get_the_category(); 
echo $category[0]->count;

In your check to see if there is only one post in the specific category and then do something, you can try

if( 1 == $category[0]->count ) {
   // Do something if the category has one post only
}