In case it’s helpful to anyone else, here’s what my code looked like to achieve this. I didn’t end up saving the attachment data at all, per tbuteler’s suggestion, but rather than using $_REQUEST I found I could use the $attachment array directly:
function my_attachment_fields_to_edit( $form_fields, $post ) {
$supported_exts = supported_types(); // array of mime types to show checkbox for
if ( in_array( $post->post_mime_type, $supported_exts ) ) {
// file is supported, show fields
$use_sc = true; // check box by default (false not to, or use other criteria)
$checked = ( $use_sc ) ? 'checked' : '';
$form_fields['use_sc'] = array(
'label' => 'Use SC',
'input' => 'html',
'html' => "<input type="checkbox" {$checked} name="attachments[{$post->ID}][use_sc]" id='attachments[{$post->ID}][use_sc]' /> " . __("Insert shortcode instead of link", "txtdomain"),
'value' => $use_sc
);
}
return $form_fields;
}
function my_media_insert( $html, $id, $attachment ) {
if ( isset( $attachment['use_sc'] ) && $attachment['use_sc'] == "on" ) {
$output="[shortcode url="".$attachment['url'].'"]';
return $output;
} else {
return $html;
}
}
// add checkbox to attachment fields
add_filter( 'attachment_fields_to_edit', 'my_attachment_fields_to_edit', null, 2 );
// insert shortcode if checkbox checked, otherwise default (link to file)
add_filter( 'media_send_to_editor', 'my_media_insert', 20, 3 );
EDIT 1/2013: This solution no longer works in WordPress 3.5+. The box-checking on the attachment screen sends an AJAX request to save the attachment metadata. There doesn’t seem to be a way to use a checkbox merely as a “flag” on the attachment screen anymore in light of this. If someone finds another solution, I will happily change the approved answer!