How to get attachment caption (get_the_excerpt gives parent post excerpt) ?

get_the_excerpt() should work for getting caption just fine.

Your issue is that it looks for post to process in global variables and in your code there you are not setting it up with attachments you are iterating through.

You need to use setup_postdata() for it to work.

Other way would be something like:

get_post_field('post_excerpt', $attachment->ID);