How do I implement a jQuery UI dialog modal in a WP plugin?

Although I’m not clear on what exactly you need, maybe I could offer a few pointers.

First off, the documentation on the various sites have everything you need, so you should visit the WordPress Codex, and jQuery UI examples.

Hooks

admin_init : You can deregister/register scripts here. If you intend on using a custom jquery ui script for example :

 function my_register_scripts()
 {
      wp_deregister_script( 'jquery-ui' ); //Deregister WP's version
      wp_register_script( 'jquery-ui', plugins_url( 'js/jquery-ui.js', __FILE__ ), array( 'jquery' ) ); //Register your own
 }
 add_action( 'admin_init', 'my_register_scripts' );

admin_menu: add your plugin page, hook specific page for your scripts also

 function my_menu()
 {
      $admin_page = add_menu_page(/*Look at the codex for the args, link below*/); //Add your page

      //use the $admin_page variable for targeted script loading

      add_action( 'admin_print_styles-' . $admin_page, 'function_that_enqueues_css_here' );
      add_action( 'admin_print_scripts-' . $admin_page, 'function_that_enqueues_js_here' );

 }
 add_action( 'admin_menu', 'my_menu' );

add_menu_page

EDIT: Dialog Example

Heres a simple modal dialog example that opens a form :

 $(document).ready(function(){
      $('#form-wrap').dialog({
           autoOpen: true, //FALSE if you open the dialog with, for example, a button click
           title: 'My Form',
           modal: true
      });
 });

The code above assumes you have or other tag with the id of ‘form-wrap’. Inside you place your form.

Folder Structure

I could be wrong, but I don’t think this matters much, I just put my .js files in a /js folder, css in a /css folder, etc…

Use plugins_url() when enqueueing files from your plugin directory.

The jQuery UI Dialog’s page should have all the examples you need.

When I use those dialogs, I generally generate html on the page (display:none), and use that for the dialog popup.

Hope some of this helped.

EDIT

Also, I think customizing your own jQuery UI script on the jQuery UI website will give you less files to enqueue.

EDIT 2

You have errors in your add_actions if the code above is what you’re using.

 add_action('admin_init', "#dialog-form" );

The second argument should be a PHP function, that looks like the ID of your HTML div.

 add_action( 'admin_enqueue_scripts-options_page-{page}', 'myplugin_admin_scripts' ); 

I’m pretty sure {page} is supposed to be replaced by the specific page you want to call the scripts on.

If jQuery is your main issue, I suggest maybe getting it working on a simple html page disassociated from your site until you have it working, then get the hooks right.