How to restrict add media library only to images of the same post family?

You need to alter the query using posts_where and posts_join filters.

function restrict_media_images_per_post_type($query) {
    add_filter('posts_where', 'media_posts_where');
    add_filter('posts_join', 'media_posts_join');
    return $query;
}
add_filter('ajax_query_attachments_args', 'restrict_media_images_per_post_type');

Change “WHERE” clause to restrict to specific post type to which selected post belongs to

function media_posts_where($where) {
    global $wpdb;
    $post_id = false;
    $whitelist_post_type = array(
        'post',
        '{custom post type}' //change this according to your need e.g. projects
    );
    if ( isset($_POST['post_id']) ) {
        $post_id = $_POST['post_id'];
        $post = get_post($post_id);
        if ( $post && in_array($post->post_type, $whitelist_post_type)) {
            $where .= $wpdb->prepare(" AND my_post_parent.post_type = %s ", $post->post_type);
            //$where .= $wpdb->prepare(" AND my_post_parent.post_type = %s AND {$wpdb->posts}.post_parent = %d", $post->post_type, $_POST['post_id']);  //Use this if you want to restrict to selected post only
        }
    }
    return $where;
}

Change “JOIN” clause to get media by restricting to the post parent

function media_posts_join($join) {
    global $wpdb;
    if ( isset($_POST['post_id']) ) {
        $join .= " LEFT JOIN {$wpdb->posts} as my_post_parent ON ({$wpdb->posts}.post_parent = my_post_parent.ID) ";
    }
    return $join;
}