You need to do an ajax call with action: ‘set-post-thumbnail’
Check in admin-ajax.php (line 1477 in 3.3.2) for the expected values and nonce, but in general you need to send post_id, attachment_id and nonce.
The nonce should come from: wp_create_nonce( “set_post_thumbnail-$post_id” );
The admin does something like:
uploader.bind('FileUploaded', function(up, file, response) {
jQuery.post(ajaxurl, {
action:"set-post-thumbnail", post_id: post_id, thumbnail_id: response.response, _ajax_nonce: '<?php echo $ajax_nonce;?>' , cookie: encodeURIComponent(document.cookie)
}, function(str){
var win = window.dialogArguments || opener || parent || top;
if ( str == '0' ) {
alert( setPostThumbnailL10n.error );
} else {
jQuery('#postimagediv .inside').html(str);
jQuery('#postimagediv .inside #plupload-upload-ui').hide();
}
}
);
jQuery("#postimagediv .inside h2.uploading_message").remove();
});