There’s no such thing as a link to a gallery, a post has a gallery by virtue of it having child attachment posts, it’s not a separate entity.
You could achieve the modal window by using some sort of lightbox plugin and modifying the gallery shortcode to output the images as just anchor links with the same rel attribute and hide them with css. Obviously that’s a bit of custom code, not sure if there’s a plugin out there that already does this.
Alternately, you could create separate gallery posts and embed a gallery from another post with the gallery shortcode, then link to those gallery posts.