WordPress Template Hierarchy

You can use the dynamic filter {type}_template (where {type} is the current query type e.g. single, index etc.) found in get_query_template():

function wpse_204848_get_id_template( $template ) {
    if ( $post = get_queried_object() ) {
        if ( $_template = locate_template( "single-{$post->post_type}-{$post->ID}.php" ) )
            $template = $_template; 
    }

    return $template;
}

add_filter( 'single_template', 'wpse_204848_get_id_template' );

For a portfolio post of ID 5, it will use the template single-portfolio-5.php if it exists.