wp_get_attachment_link() add rel attribute if the link points to direct image

add_filter( 'wp_get_attachment_link', 'sant_prettyadd', 10, 6);

function sant_prettyadd ($content, $id, $size, $permalink, $icon, $text) {
    if ($permalink) {
    return $content;    
    }
    $content = preg_replace("/<a/","<a rel=\"prettyPhoto[slides]\"",$content,1);
    return $content;
}

Update:

function sant_prettyadd checks permalink argument.

if the permalink = true then it returns the content as it is.

if the permalink = false then it skips to the next line and replace the <a with <a rel="prettyPhoto[slides]" and then returns the content.