Is “get_template_part” hierarchy possible?

Yes, what you describe is already built in to the get_template_part() function:

<?php get_template_part( $slug, $name ); ?> 

Where “masthead” is the general template slug in your example, and “custom” is the more specialized version. You’ll have to name your parts slug-name.php (so, masthead-custom.php, rather than custom-masthead.php).

Note that you can also use the value returned by a function for the $name argument — I do this all the time for things like:

get_template_part( 'content', get_post_type() );

In all cases, if the specialized template part isn’t found, masthead.php is used as a fallback.