Is there a hook to put stylesheet and/or JS inside iframes (thickbox or tinyMCE) in admin area

You could over-ride the CSS by using the admin_print_scripts admin_head-media-upload-popup and add css to match your needs. This can be done via the functions.php file or by creating a plugin. Here is the code in a plugin format to begin adding style:

<?php
/*
Plugin Name: Some Name
Description: Custom Thickbox Styles
*/
add_action('admin_head-media-upload-popup', 'custom_tb_styles');
function custom_tb_styles() {
  ?>
  <style>
    #TB_window {
    background:silver;
    }
    /*
    YOUR CUSTOM STYLES HERE
    */
  </style>
  <?php
  }

If you’re adding the code to your functions.php file you could just add this to the file:

<?php
add_action('admin_head-media-upload-popup', 'custom_tb_styles');
function custom_tb_styles() {
  ?>
  <style>
    #TB_window {
    background:silver;
    }
    /*
    YOUR CUSTOM STYLES HERE
    */
  </style>
  <?php
  }

Another option is to unregister the Thickbox Styles and completely add your own.

_Edited according to the comment by ungestaltbar, which points to the correct hook to print the style/script only in the thickbox iframe. The action hooks used earlier (admin_print_scripts & admin_print_styles) will be called in all administrative pages.