Infinite next / prev post in single.php page

I think this is a better and cleaner approach of what you are trying to achive.

// get_adjacent_post will return $post object
$prev_post = get_adjacent_post(true,'',true);
//Using get_the_post_thumbnail() to get featured image of previous post
$prev_thumbnail = get_the_post_thumbnail( $prev_post->ID, array( 100, 100));

$next_post = get_adjacent_post(true,'',false);
//Using get_the_post_thumbnail() to get featured image of next post
$next_thumbnail = get_the_post_thumbnail( $next_post->ID, array( 100, 100));

the_post_navigation(
    array(
        'in_same_term' => true,
        'next_text' => '<p class="meta-nav">'. $next_thumbnail .'Next Post &rarr;</p><p class="post-title">%title</p>',
        'prev_text' => '<p class="meta-nav">'. $prev_thumbnail .'&larr; Previous Post</p><p class="post-title">%title</p>',
    )
);

Okay the solution above will require additional next_post_link filter and prev_post_link filter and overriding the_post_navigation template which is not a realy good solution. So based on your code try this one. I have tested 3 posts from same category and its looping without problems.

$prev_post = get_adjacent_post(true,'',true);
if($prev_post):
    $prev_post_id = $prev_post->ID;
    $prev_thumbnail = get_the_post_thumbnail($prev_post_id, array( 100, 100));
else:
    $abjacent_categories = get_the_category(); 
    $ids = array();
    foreach($abjacent_categories as $category) {
        $ids[] = $category->term_id;
    }
    $last = new WP_Query( array( 
        'post_type' => 'post',
        'posts_per_page' => 1, 
        'orderby'         => 'post_date',
        'order'           => 'DESC',
        'category__in' => $ids, 
/* use category__and if you want post to be in all categories as current post. 
For now its happy with one of current post categories. 
Depending on this will give different results at the end so keep that in mind. */
     ) ); 
    $last->the_post(); 
    $prev_post_id = get_the_ID();
    $prev_thumbnail = get_the_post_thumbnail( $prev_post_id, array( 100, 100));
wp_reset_query();
endif;

$next_post = get_adjacent_post(true,'',false);
if($next_post):
    $next_post_id = $next_post->ID;
    $next_thumbnail = get_the_post_thumbnail($next_post_id, array( 100, 100));
else:
    $abjacent_categories = get_the_category(); 
    $ids = array();
    foreach($abjacent_categories as $category) {
        $ids[] = $category->term_id;
    }
    $first = new WP_Query( array( 
        'post_type' => 'post',
        'posts_per_page' => 1, 
        'orderby'         => 'post_date',
        'order'           => 'ASC',
        'category__in' => $ids,     
/* use category__and if you want post to be in all categories as current post. 
For now its happy with one of current post categories. 
Depending on this will give different results at the end so keep that in mind. */
     ) ); 
    $first->the_post(); 
    $next_post_id = get_the_ID();
    $next_thumbnail = get_the_post_thumbnail( $next_post_id, array( 100, 100));
wp_reset_query();
endif;

For the html you can test with following template

<div class="nav-links">
    <div class="nav-previous">
        <a href="<?php echo get_the_permalink($prev_post_id); ?>" rel="prev">
            <p class="meta-nav"><?php echo $prev_thumbnail ?> &larr; Previous Post</p><p class="post-title"><?php echo get_the_title($prev_post_id); ?></p>
        </a>
    </div>
    <div class="nav-next">
        <a href="<?php echo get_the_permalink($next_post_id); ?>" rel="next">
            <p class="meta-nav"><?php echo $next_thumbnail ?> Next Post &rarr;</p><p class="post-title"><?php echo get_the_title($next_post_id); ?></p>
        </a>
    </div>
</div>