jQuery UI styles conflicting with TinyMCE dialog

It should work just fine if you enqueue your script & stylesheet within functions.php file.

This is how I do that (feel free to copy & paste, it should work out of the box):

function load_my_admin_js() {
   wp_register_script('jquery-ui',"https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.11/jquery-ui.min.js");
   wp_enqueue_script('jquery-ui');   
} 

add_action('admin_init', 'load_my_admin_js');   

function load_my_admin_css() {
   wp_enqueue_style( 'jquery-ui', 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8/themes/base/jquery-ui.css');  
} 

add_action( 'admin_print_styles', 'load_my_admin_css');  

Don’t forget to delete your previous links to jQuery UI.

If something is still wrong, then your custom JS code is corrupted and affects other elements. Can’t say anything more without investigating the code itself.