So i finally figured it out, the gallery switches modes sometimes, even without clicking anything (not completely sure what actually happens, I have no idea about Backbone tbh.)
Here is what now works for me:
if(wp.media.frame.content.get('gallery').collection) {
wp.media.frame.content.get('gallery').collection.add(formatted);
} else {
wp.media.frame.controller.content.get('gallery').collection.add(formatted);
}
The else
-line won’t work when initially loading of media library, so I check before and use the first line if I can. So far it seems to work in all situations (including in media-modal when assigning an image to a custom post-type)