custom post type taxonomies UI radiobuttons not checkboxes

When you register a taxonomy WordPress automatically handles producing the appropriate metabox. First you need to ‘de-register’ this default metabox:

add_action( 'admin_menu', 'myprefix_remove_meta_box');  
function myprefix_remove_meta_box(){  
   remove_meta_box('my-tax-metabox-id', 'post', 'normal');  
}

Where my-tax-metabox-id is the ID of your metabox. Then ‘re-register’ the metabox providing your own callback function which produces in the output:

//Add new taxonomy meta box  
 add_action( 'add_meta_boxes', 'myprefix_add_meta_box');  
 function myprefix_add_meta_box() {  
     add_meta_box( 'mytaxonomymetabox_id', 'My Taxonomy','myprefix_mytaxonomy_metabox','post' ,'side','core');  
 } 

 function myprefix_mytaxonomy_metabox( $post ) {  
     //This function determines what displays in your metabox
     echo 'This is my taxonomy metabox';  
  } 

Then it is simply a matter of mimicking the ‘default’ hierarchal metabox markup but altering the checkboxes to radio buttons. The function that is responsible for producing the default mark-up can be found here: http://core.trac.wordpress.org/browser/tags/3.3/wp-admin/includes/meta-boxes.php#L307

I show in detail how to do this in this article I wrote. You may also find this corresponding repository helpful.