It’s because you are have passed “category” in “get_terms_by()” function as third argument.
Please replace “add_custom_taxonomy_here” with your custom taxonomy slug in below code.
// Add Shortcode to show posts count inside a category
function category_post_count( $atts ) {
$atts = shortcode_atts( array(
'category' => null
), $atts );
// get the category by slug.
$term = get_term_by( 'slug', $atts['category'], 'add_custom_taxonomy_here');
return ( isset( $term->count ) ) ? $term->count : 0;
}
add_shortcode( 'category_post_count', 'category_post_count' );