it’s quite easy:
$months = $wpdb->get_col("SELECT DATE_FORMAT(post_date_gmt, '%m')
FROM $wpdb->posts
LEFT JOIN $wpdb->term_relationships as t
ON ID = t.object_id
WHERE post_type="post" AND post_status="publish" AND t.term_taxonomy_id = 11
AND DATE_FORMAT(post_date_gmt, '%Y') = $y
GROUP BY DATE_FORMAT(post_date_gmt, '%Y-%m')
ORDER BY post_date_gmt DESC");
term_taxonomy_id is the category. And I join on the object ID