Redirecting specific category posts to subdomain

You can use template_redirect action hook to achieve that:

add_action('template_redirect', 'redirect_category_posts');
function redirect_category_posts() {
    global $post;
    if (is_single($post->ID) && in_category('category-name', $post)) {
        $new_url = "http://category-name.mydomain.com/{$post->post_name}.html";
        wp_redirect($new_url, 301);
        exit;
    }
}

Please note that both posts in main domain and in sub-domain must have the same slug or post_name for this code to work.