Taxonomy Parent not showing up in address bar

I found an answer within 10 minutes of posting this. 🙂 Just add 'hierarchical' => true in the rewrite=>array() of the taxonomy as follows:

$args = array( 
    ...
    'hierarchical'  => true,
    'rewrite'       => array( 
        'slug'          => 'THE-CHOSEN-SLUG',
        'hierarchical'  => true,
    ),
);
register_taxonomy( 'my-custom-taxonomy', array( 'page', 'my-custom-post-type' ), $args );

Then flush the rewrite rules by going to the Permalink settings and saving them, and voila!