In short: WordPress uses the term count in various places. There’s even another count that it’s in the options table. So as long as you are talking about the core behavior, then the recommendation is:
Never ever touch the WordPress cores term counting mechanism!
It’s hard enough to bring it to work in some cases (manually updating term stuff in very, very custom scenarios for e.g.).
If your problem is that it is used somewhere in the public (like user admin pages or theme templates), then simply add your own counter in parallel or remove the displaying function at all.