WordPress Custom Post Loop

First of all why do you call wp_reset_query() on each while iteration?
Then we simply will create new $cats array and check if new “category” is not already there = then show and add it to the array.

  <ul class="filter filter-top">
    <li class="active"><a href="#" data-filter="*">All</a></li>
    <?php
      $portfolio_args_custom = array(
        'post_type' => 'portfolio',
      );
      $cats = [];
      $portfolio_posts_custom = new WP_Query($portfolio_args_custom);
      while($portfolio_posts_custom->have_posts()) {
        $portfolio_posts_custom->the_post();
        $cat = get_post_meta( get_the_ID(), 'Category', true);
        if (!in_array($cat, $cats)) {
           $cats[] = $cat;
   ?>
     <li><a href="#" data-filter=".<?php echo $cat;?>"><?php echo $cat;?></a></li>
      <?php } } wp_reset_query(); ?>
  </ul>