WordPress shortcode Issue!

Two issues here

• Never use extract(). It has been removed from core functions, that should tell you a lot. For more info, check this trac ticket #22400

• Your tax_query is incorrect, it should be an array of an array, yours is just an array

Your shortcode should look something like this

function course_listings( $atts ) 
{
    // Attributes
    $defaults = shortcode_atts( array(
        'course_category' => '',
    ), $atts );

    // Code

    $args = array(
        'post_type' => 'modules',
        'tax_query' => array(
            array(
                'taxonomy' => 'courses',
                'field'    => 'slug',
                 'terms'   => $defaults['course_category'],
            ),   
        ),
    );

    $mod_query = new WP_Query( $args );

    $output="<div class="course-listings">";

    while ( $mod_query->have_posts() ) {

        $mod_query->the_post();

        $output .='<ul>';
        $output .='<li>' .get_the_title(). '</li>';
        $output .='</ul>';

    }

    wp_reset_postdata();
    $output .='</div>';

    return $output;
}
add_shortcode( 'course', 'course_listings' );

Just a few tip, you would most probably first want to check if you have a term entered before executing the query