the_excerpt filter doesn’t work as expected

Generating the excerpt is quite a jumble of filters, so it’s difficult to see precisely what is happening, but here is my guess:

WordPress attaches a default filter (see line 148) to get_the_excerpt, namely wp_trim_excerpt. As you can see from the latter function’s source code, it starts with getting the content if it gets passed an empty string. Now, if this filter is executed after yours and there are only shortcodes in the post, your filter will pass an empty string and the default filter will simply ignore everything you have done and start generating its own excerpt. So you could try making sure your filter is executed later, by changing its priority:

add_filter ('the_excerpt', 'my_custom_excerpt', 1, 20);

Alternatively, you could remove that filter:

remove_filter('get_the_excerpt', 'wp_trim_excerpt', 10);