When you register your taxonomy, set the slug
parameter of rewrite
to your desired structure:
register_taxonomy(
'custom_tax',
'custom_post_type',
array(
'rewrite' => array( 'slug' => 'your-cpt-slug/category' ),
// your other args...
)
);