I asked this in the WP Contributors Slack and was pointed towards this code that allows you to specify the ID of the media library you want to pull from.
https://github.com/humanmade/network-media-library
This doesn’t exactly do what I want, it sets a single media library for everysite to use, rather than a specific site on command.