One simple option would be to add a parameter to the query string of the portfolio item (i.e. yousite.com/portfolio-item/?backto=portfolio-3), then on the item page modify the ‘back’ link with JavaScript: if the backto parameter exists then change the link to point to the portfolio specified in it.
Here’s some untested starter code. On the portfolio page (assuming you’re looping through portfolio items and printing links to their pages):
<?php $portfolio_name="whatever"; ?>
<a href="https://wordpress.stackexchange.com/questions/138758/<?php echo get_permalink() ."?backto=' . $portfolio_name; ?>"><?php the_title(); ?></a>
On the portfolio page:
$key_backto = (isset($_GET['backto']) ) ? $_GET['backto'] : '';
<a href="https://wordpress.stackexchange.com/questions/138758/<?php echo home_url("portfolio/' . $key_backto); ?>">Back to portfolio</a>