I think you are a bit confused as to what Page Attributes will give you access to. To the best of my knowledge you do not get a drop down to choose a template (as you do on Pages) as an automatic function on custom post types.
That being said, you are on the right track. You need to make an archive-portfolio.php file as well to display all of the portfolio posts as well.
You do not need an additional WP_Query to pull in your correct post type – WordPress will do that correctly based on single-POSTTYPE.php.
Ditch that line and revert to the standard loop – and ditch the template name as well. It does not do anything. If you really, really, really want to use this as a Page, rename the template page-portfolio.php and create a page, select this template and you’ll be off like a shot.