If you are never going to use the attachment page, then you can simply change the HTTP header ( a php function ) of attachment page to the image file.
Example –
header('Location: ' . wp_get_attachment_url() );
Note – Call it inside WordPress loop.