I want to rel=nofollow wordpress images

You need to hook into the image_send_to_editor filter. This allows you to modify the markup used when inserting an image into the content editor.

Something like this should work:

add_filter('image_send_to_editor', 'my_add_rel_nofollow', 10, 8);
function my_add_rel_nofollow($html, $id, $caption, $title, $align, $url, $size, $alt="" ){

    // check if there is already a rel value
    if ( preg_match('/<a.*? rel=".*?">/', $html) ) {
        $html = preg_replace('/(<a.*? rel=".*?)(".*?>)/', '$1 ' . 'nofollow' . '$2', $html);
    } else {
        $html = preg_replace('/(<a.*?)>/', '$1 rel="' . 'nofollow' . '" >', $html);
    }
    return $html;
}

This will first check if there is already a rel attribute, and add the no_follow. Or, if there is no rel attribute, it will add one, and set it to nofollow.