How do I modify the “Insert Media” lightbox in the admin to only show media items from a category?

I think you can go about the filter list_terms_exclusions, but i don`t tested.

example:

function fb_list_terms_exclusions($args) {

    if ( 'media-upload-popup' != $where )
        return $where;

    if ( !current_user_can('manage_categories') )
        $where = " AND t.slug  NOT IN ('uncategorized', 'category-2')"; // slug of categories

    return $where;
}
add_filter( 'list_terms_exclusions', 'fb_list_terms_exclusions' );

update for the comments to this asnwer: an example to find the post types; at first a small example to use the default variables and after this an funtion for return an value:

/**
 * Include my scripts
 *
 * $pagehook for check, if this the right page
 * $post_type for check, if this the right post type
 */
function fb_post_type_script($pagehook) {
    global $post_type;

    $pages = array( 'edit.php', 'post.php', 'post-new.php' );
    if ( in_array( $pagehook, $pages ) && $post_type == 'my_post_type' ) {
        wp_enqueue_script( 'my_script_example_key', plugins_url( 'js/my_example_script.js', __FILE__ ), array( 'jquery', 'my_other_example_script_key' ), '1.0.0' );
    }
}
add_action( 'admin_enqueue_scripts', 'fb_post_type_script' );

alternative you use a custom function to find the post type:

/**
 * Return post type
 *
 * @return string $post_type
 */
private function get_post_type() {

    if ( !function_exists('get_post_type_object') )
        return NULL;

    if ( isset($_GET['post']) )
        $post_id = (int) $_GET['post'];
    elseif ( isset($_POST['post_ID']) )
        $post_id = (int) $_POST['post_ID'];
    else
        $post_id = 0;

    $post = NULL;
    $post_type_object = NULL;
    $post_type = NULL;
    if ( $post_id ) {
        $post = get_post($post_id);
        if ( $post ) {
        $post_type_object = get_post_type_object($post->post_type);
            if ( $post_type_object ) {
                $post_type = $post->post_type;
                $current_screen->post_type = $post->post_type;
                $current_screen->id = $current_screen->post_type;
            }
        }
    } elseif ( isset($_POST['post_type']) ) {
        $post_type_object = get_post_type_object($_POST['post_type']);
        if ( $post_type_object ) {
            $post_type = $post_type_object->name;
            $current_screen->post_type = $post_type;
            $current_screen->id = $current_screen->post_type;
        }
    } elseif ( isset($_SERVER['QUERY_STRING']) ) {
        $post_type = esc_attr( $_SERVER['QUERY_STRING'] );
        $post_type = str_replace( 'post_type=", "', $post_type );
    }

    return $post_type;
}

public function example() {
    $post_type = $this->get_post_type();  

    // FB_POST_TYPE_1 is defined with my post type
    if ( FB_POST_TYPE_1 == $post_type ) {
        // tue etwas, wenn wahr
    }
}