Prevent duplicate post from dynamic multiple loops

Don’t run a new query for your CPT at every iteration of the main Loop. Pull it once, outside the Look with 'posts_per_page' => 4 then increment it as you go.

In other words:

$counter = 1;
$loop = new WP_Query( 
  array( 
    'post_type' => 'post', 
    'posts_per_page' => 20
  ) 
);

$inner_query = new WP_Query(
  array( 
    'post_type' => 'page', 
    'posts_per_page' => 4, 
    'orderby' => 'rand'
  )
);

while ( $loop->have_posts() ) {
  $loop->the_post(); ?>
  <li class="group"><?php
    the_title(); 
    echo '('.$post->post_type.')'; ?>
  </li><?php
  if ($counter == 5) {
    if ($inner_query->have_posts()) {
      $inner_query->the_post(); ?>
      <li><?php
        the_title(); 
        echo '('.$post->post_type.')'; ?>
      </li><?php
    }
    $counter = 0;
  }
  $counter++ ;
}

You have to be very careful with counters and think through the logic or you will get unexpected results.

Secondly, I simplified your logic. There is no need for the mod math if you use a counter as you do. However, there is a way to do away with the counter altogether, as WP_Query provides one for you.

while ( $loop->have_posts() ) { 
  $loop->the_post(); ?>
  <li class="group"><?php
    the_title(); 
    echo '('.$post->post_type.') ('.$loop->current_post.')'; ?>
  </li><?php
  if ($loop->current_post != 0 && ($loop->current_post+1) % 5 == 0) {
    if ($inner_query->have_posts()) {
      $inner_query->the_post(); ?>
      <li><?php
        the_title(); 
        echo '('.$post->post_type.')'; ?>
      </li><?php
    }
  }
}