You are looking at the wrong line. $this_taxonomy is telling what taxonomy type (category, tag, custom tax, etc.) to look for, not what actual term in that taxonomy type.
That being said, you should look at $previous and $next. More specifically, get_previous_post_link() and get_next_post_link(). The fourth parameter is $excluded_terms which is currently empty.
The codex says you can supply an array or comma-separated list of term ids.
This would seem cumbersome to maintain because you would have to update that parameter each time a new term is added to your taxonomy.
But technically, if you have categories 1,2,3,4 and 5, then supplying $excluded_terms="4,5"; to your previous and next function would exclude those terms from your list.
So, if you want a method that won’t need updating each time a new category is added, I would try to do it by modifying the main query instead. Add this to your functions.php
add_action( 'pre_get_posts', 'my_allowed_cats' );
function my_allowed_cats( $query ){
$post_type = get_post_type();
if( $post_type == 'post' || $post_type == 'portfolio' ){
// Use category ids
$allowed_cats = array(
1,
2,
3,
);
$query->set( 'category__in', $allowed_cats );
}
}
This is untested, but it should work