You have to use the Slug of your Custom Post Type “portfolio” for the filename, as you pointed out, single-portfolio.php is correct if your slug is “portfolio”.
You just have to take one more step, create the file in the Theme directory (the same folder where your single.php is located).
WordPress automatically selects the single-portfolio.php for the Portfolio CPT if the file is there, and if it is not there, it takes the single.php.