You have inserted wrong thumb functions for large fancybox image URLs.
<a href="https://wordpress.stackexchange.com/questions/159324/<?php get_the_post_thumbnail(); ?>" class="project-img fancybox"></a>
This get_the_post_thumbnail() returns an HTML image element but you need URL of image.
Your first code should be like this.
<?php $args = array( 'post_type' => 'portfolio' );
$loop = new WP_Query( $args );
while ( $loop->have_posts() ) : $loop->the_post();
$thumb_image_url = wp_get_attachment_image_src( get_post_thumbnail_id( $post->ID ), 'large' ); ?>
<div class="portfolio-item">
<a class="project-img-container">
<?php the_post_thumbnail(); ?>
</a>
<div class="overlay">
<a href="https://wordpress.stackexchange.com/questions/159324/<?php the_permalink(); ?>" class="project-title"><?php the_title(); ?></a>
<a href="<?php echo $thumb_image_url[0]; ?>" class="project-img fancybox"></a>
<a href="https://wordpress.stackexchange.com/questions/159324/<?php the_permalink(); ?>" class="project-link"></a>
</div>
</div>
<?php endwhile; ?>