WordPress Admin: open popup window on a custom button

Maybe you can open a thickbox with javascript.

A example with check for the height and width of the thickbox.

                    <script type="text/javascript">
                    <!--
                        var viewportwidth,
                            viewportheight;
                        if (typeof window.innerWidth != 'undefined' ) {
                            viewportwidth = window.innerWidth-80,
                            viewportheight = window.innerHeight-100
                        } else if (typeof document.documentElement != 'undefined'
                            && typeof document.documentElement.clientWidth !=
                            'undefined' && document.documentElement.clientWidth != 0)
                        {
                            viewportwidth = document.documentElement.clientWidth,
                            viewportheight = document.documentElement.clientHeight
                        } else { // older versions of IE
                            viewportwidth = document.getElementsByTagName('body' )[0].clientWidth,
                            viewportheight = document.getElementsByTagName('body' )[0].clientHeight
                        }
                        document.write('<a onclick="return false;" href="https://wordpress.stackexchange.com/questions/96643/<?php echo WP_PLUGIN_URL ."https://wordpress.stackexchange.com/" 
                            . FB_WM_BASEDIR; ?>/index.php?KeepThis=true&amp;TB_iframe=true&amp;height=" 
                            + viewportheight + "&amp;width=" + viewportwidth 
                            + "&amp;modal=false" class="thickbox button"><?php _e( 'Preview', FB_WM_TEXTDOMAIN ); ?></a>' );
                        //-->
                    </script>

Also it is important, that you initialize all requirements, via add_thickbox();. Inside the Thickbox you can load your parent data.