function my_portfolio_meta_box_cb($post) {
$template = get_post_meta($post->ID, '_wp_page_template', true);
if ($template == 'portfolio.php') {
add_meta_box(........);
}
}
Hook this via the register_meta_box_cb
arg when registering the post type or in your manually via add_meta_boxes-portfolio
hook.