WordPress 3.5 attachment_fields_to_edit and media_send_to_editor

So, there are some things which work differently (now).

I. What you’ve noticed, when you click the checkbox WordPress sends an Ajax request to ‘wp_ajax_save_attachment_compat()’ with some form data, if your checkbox is checked, this information will be part of the form data.

‘wp_ajax_save_attachment_compat()’ checks for $_REQUEST[‘attachments’] and $_REQUEST[‘attachments’][$id] and will fail if it does not exist.

This means you’ll have to change your input field to:

<input type="checkbox" name="attachments[$post->ID][ab_prettyphoto]" id='ab_prettyphoto-{$post->ID}' value="1" />

II. Now this will pass all checks in ‘wp_ajax_save_attachment_compat’ and arrives at a useful filter:
(this is line 1930 in ajax-actions.php)

$post = apply_filters( 'attachment_fields_to_save', $post, $attachment_data );

You can use this filter to add your field data as post meta to the attachment, later on, you will check if this post meta exist and do your action if true:

add_filter('attachment_fields_to_save', 'wpse76219_add_meta',10,2);
function wpse76219_add_meta($post, $attachment_data){

    // use this filter to add post meta if key exists or delete it if not
    if ( !empty($attachment_data['ab_prettyphoto']) && $attachment_data['ab_prettyphoto'] == '1' )
         update_post_meta($post['ID'], 'ab_prettyphoto', true);
    else
         delete_post_meta($post['ID'], 'ab_prettyphoto');

    // return $post in any case, things will break otherwise
    return $post;
}

III. Last step, use ‘media_send_to_editor’ to check if your meta data exists, if true, do string manipulation, else return original:

add_filter('media_send_to_editor', 'wpse76219_send_to_editor', 10,3);
function wpse76219_send_to_editor( $html, $id, $att)
{
    $is_set = get_post_meta($id,'ab_prettyphoto', true);
    if ($is_set and $is_set == '1')
        return str_replace('<a', '<a rel="prettyPhoto-img"', $html);
    else
        return $html;
}