How to remove this filter?

It depends on the call sequence. Are you sure that the remove_filter is called after the add_filter and before the attachment_fields_to_edit filter is invoked? Add few trace statements and verify.