As far as I remember counts for terms are stored in database, so there is nothing to modify when you fetch them – you simply get ready-made numbers.
So you will either need to implement and maintain your special logic for counts completely separately or try to recalculate and modify native counts, see wp_update_term_count_now().