order posts by a secondary query that counts items

The way to go is apply custom filters to the WP_Query you are running.

add_filter('posts_clauses', function ($clauses, $query) {
    global $wpdb;

    // only run the filter for a given query
    // do your checks and return early
    if (!$query->is_main_query() || !$query->is_post_type_archive('tracklist')) {
        return $clauses;
    }

    $clauses['fields'] .= ", COUNT(items.id) AS track_count";
    $clauses['join'] .= " LEFT JOIN tracklist_items AS items ON {$wpdb->posts}.ID = items.tracklist_id";
    $clauses['groupby'] = "{$wpdb->posts}.ID";
    $clauses['orderby'] = "track_count DESC";

    return $clauses;
}, 2, 10);

See also https://stackoverflow.com/questions/20119901/counting-number-of-joined-rows-in-left-join