How can I remove the image attachment ALT field?

Considering that you will come up with a solution to the accessibility issue pointed by @toscho…

Putting a lower priority (later execution) to the filter does the job.

$priority (integer) (optional)
Used to specify the order in which the
functions associated with a particular action are executed. Lower
numbers correspond with earlier execution, and functions with the same
priority are executed in the order in which they were added to the
action.
Default: 10

function remove_caption( $fields ) {
    unset( $fields['image_alt'] );
    return $fields;
}
add_filter( 'attachment_fields_to_edit', 'remove_caption', 999, 1 );