If all you want to do is count how many times the shortcode exists in the content then you can use substr_count on the original content before it’s parsed to output shortcodes like this:
$count = substr_count( get_the_content(), '[col]' );
But nesting shortcodes is most likely the best solution unless you know exactly the type of content being added to the post/page.
So your full code would look like this:
// Get the content
$content = get_the_content();
// Get number of col shortcodes added to the content
$count = substr_count( $content, '[col]' );
// Output content
if ( $count > 1 ) {
return '<div class="multi_columns">' . do_shortcode( $content ) . '</div>';
} else {
return '<div class="column">' . do_shortcode( $content ) . '</div>';
}
But I see no reason to avoid using nested columns. Especially if this is for a client site or an item for reproduction if someday they want to change the way the site works it could potentially issues – it seems the way you want things to work expects only columns to be added to the post content (but maybe that’s what you want?). In which case you may be better off not using column shortcodes and instead using a repeatable meta field such as this – https://www.advancedcustomfields.com/add-ons/repeater-field/