I’m pretty sure you could use JavaScript to do this for you very simply. I’d try this out:
function convert_root_cats_to_radio()
{
global $post_type;
if ( 'post' != $post_type )
return;
?>
<script type="text/javascript">
jQuery("#categorychecklist>li>label input").each(function(){
this.type="radio";
});
</script>
<?php
}
add_action( 'admin_footer-post.php', 'convert_root_cats_to_radio' );
add_action( 'admin_footer-post-new.php', 'convert_root_cats_to_radio' );
What this does is loops through the hierarchy and converts the top level items to radio buttons. If you need to strictly enforce this, the javascript won’t be a good solution because the root-level categories can still be selected under the popular tab.
Check out this plugin, too, to keep the hierarchy after saving: http://wordpress.org/extend/plugins/category-checklist-tree/
If you need something a little more enforceable, you’ll need a custom plugin to that. The above mentioned plugin would event be a great place to start in that endeavor!
Cheers~