So, I would refactor this a bit:
<?php
$featured = ( has_post_thumbnail() ? get_post_thumbnail() : false );
$first_img = false;
$custom = false;
if ( ! $featured ) {
// No featured image; look for attached images and get the first one
$attachments = get_children( 'post_type=attachment&orderby=menu_order&post_mime_type=image&post_parent=".$post->ID );
$first_img = ( $attachments ? wp_get_attachment_image_src($attachments[0]->ID, "thumbnail-size') : false );
// No attached images; look for custom-field image
if ( ! $first_img ) {
$custom_meta = ( get_post_custom($post->ID) ? get_post_custom($post->ID) : false );
$custom = ( isset( $custom_meta['custom_field_name'][0] ) ? $custom['custom_field_name'][0] : "false" );
}
}
?>
So, now, you can just step through each source; e.g.:
<?php
if ( $featured ) {
// Post has a featured image; do something with it
} else if ( $first_img ) {
// Post has attached image; do something with the first one
} else if ( $custom ) {
// Post has image attached via custom field; do something with it
} else {
// Otherwise, there's no featured image,
// no attached images, and no image
// defined in post custom meta. Do something.
}
?>
You just need to fit in your markup accordingly.