Add gallery id to rel attribe of wp_get_attachment_link

That filter passes more arguments than you are using.

return apply_filters( 'wp_get_attachment_link"https://wordpress.stackexchange.com/questions/134811/,"<a href="https://wordpress.stackexchange.com/questions/134811/$url">$link_text</a>", $id, $size, $permalink, $icon, $text );

The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your add_filter to ask for it by setting the fourth argument to 2.

function add_rel_to_gallery($link, $id) {
  return str_replace('<a href="https://wordpress.stackexchange.com/questions/134811/,"<a data-fancybox="true" rel="group-'.$id.'" href=", $link);
}
add_filter( "wp_get_attachment_link"https://wordpress.stackexchange.com/questions/134811/,"add_rel_to_gallery', 10, 2 );

After seeing your edit, I realized that I misread what you were attempting and I tried to find a way to create a static gallery specific ID without abusing hooks too much. I am not sure I succeeded in that last part but the following seems to work:

function get_id($inc = false) {
  static $id;
  if ($inc) {
    $id++;
  }
  return $id;
}

function replace($link) {
  $id = get_id();
  return str_replace('<a href="https://wordpress.stackexchange.com/questions/134811/,"<a data-fancybox="true" rel="group-'.$id.'" href=", $link);
}

add_filter(
  "post_gallery',
  function() {
    get_id(true);
    add_filter('wp_get_attachment_link','replace');
  }
);

The correct answer is probably to use the post_gallery hook to hijack the rest of the gallery shortcode, or just overwrite the gallery shortcode with one of your own.