How can I convert some categories to a custom taxonomy?

You can create whatever new taxonomies you want to by adding register_taxonomy() function in functions.php

After that you can simply assign all the posts related to Category C to that custom taxonomy using Convert Post types plugin.