How to define the template priority between built-in categories and custom taxonomies?

You can tell WordPress what template file to use with the template_redirect hook.

add_action( 'template_redirect', 'wpse98738_taxonomy_redirect' );
function wpse98738_taxonomy_redirect() {
    if( 'x' == $customtax && 'y' == $category_name ) {
        include( get_template_directory() . '/taxonomy-customtax.php' );
        // if you're using a Child Theme, use the following line instead:
        // include( get_stylesheet_directory() . '/taxonomy-customtax.php' );
        exit; // so that WP quits looking for templates to load
    }
}

Reference

Codex pages for:

template_redirect

get_template_directory() /
get_stylesheet_directory()