wp_get_attachment_image_src() regenerate custom size images everytime

No, the wp_get_attachment_image_src function calls image_downsize, which in turn calls image_get_intermediate_size itself. It doesn’t cause generation of the images, it is simply getting the relevant images from the meta and choosing the correct one to use.