Is it possible to make tag archive page specific to Custom Post Type?

If I understood correctly, you want a archive template for terms of core tag taxonomy that includes only your portfolio custom post type. The best way is to use the pre_get_posts action hook to set post_type argument of the query to 'portfolio':

add_action('pre_get_posts', 'query_post_type');
function query_post_type($query) {
   //Limit to main query, tag queries and frontend
   if($query->is_main_query() && !is_admin() && $query->is_tag ) {

        $query->set( 'post_type', 'portfolio' );

   }

}

Then, any view of tag archive will include only your custom post type without need of a secondary WP_Query. Then, you can use any of the archive templates to display the results. Obviously, the standard post are not included.

I you want to limit this only for specific tag terms:

add_action('pre_get_posts', 'query_post_type');
function query_post_type($query) {
   //Limit to main query, tag queries and frontend
   if($query->is_main_query() && !is_admin() && $query->is_tag && $query->get('tag') == 'photo' ) {

        $query->set( 'post_type', 'portfolio' );

   }

}

About use specific template for a tag archive based on a custom post type, you can’t do it directly. But uou can use, for example, tag-photo.php as template file for the archive of photo tag. If you need to cover several tags and want to avoid creating several files with the same content, you can use the template_include filter. For example, create “portfolio_archive_template.php” file, put it your theme folder and:

add_filter( 'template_include', 'portfolio_tag_template' );

function portfolio_tag_template( $template ) {
    $portfolio_tag = array( 'photo', 'video' );
    if ( is_tag() && in_array( get_query_var( 'tag' ), $portfolio_tag )  ) {
        $new_template = locate_template( array( 'portfolio_tag_template.php' ) );
        if ( '' != $new_template ) {
            return $new_template ;
        }
    }

    return $template;
}

Anyway, I think what you are trying to do is against the concept of a taxonomy. Why tag a post (of any type) with a taxonomy term but exlude it from the term archive? It is like having “yellow cats” and “yellow dogs” but exclude “yellow dogs” from “yellow animals” archive.

From my point of view is definetly a incorrect approach. Instead, you should use a custom taxonomy associated exclusively with your custom post type; for example the ‘portfolio_category’ taxonomy you have already registered. Then, you can use taxonomy-portfolio_category.php template to customize the output. As portfolio_category is a exclusive taxonomy of portfolio items, you will have the correct archive page you want without any extra code or workarounds.