Using single.php with conditional template part inclusion for displaying a specific layout for your custom post type is not the right way to do things.
WordPress has something called the Template Hierarchy. The right way would be to create a single-name-of-your-cpt.php to display single portofolio items. The same applies to custom post type archives.
The second problem is that categories are a build-in taxonomy. Category archive pages use the archive.php template, but you won’t see your custom post type show up there, because only posts are included. You could always modify the query via the pre_get_posts action, but I don’t think that this is what you want.
For your case the best thing might be to use a Custom Taxonomy.
Let’s say that you have a custom post type called portfolio and a custom taxonomy called portfolio-category.
With this set up you could use an archive-portfolio.php template to display all your portfolio posts and an taxonomy-portfolio-category.php to display the portfolio posts in a certain portfolio category. You can even create archive templates for certain portfolio categories this way.
For the single views, use single-portfolio.php and use has_term() to check in what portfolio category the post is and include the corresponding template part this way, making sure to have a default layout as fallback.