Is there a good way to load tiny_mce_popup.js through editor.windowManager.open?

You could use more appropriate path, like:

$path = ABSPATH . WPINC . '/js/tinymce/tiny_mce_popup.js'

And you should use wp_enqueue_script, so you could put something like this:

wp_enqueue_script('my-tinymce-popup', $path);

File not found.