Can I restrict category availability?

Either the taxonomy is common, or it is not. In other words, you may have books and movies CPTs that have an author taxonomy for their authors (lets say screen play in case of movies, to make it real). In the real world there is no much sense to say that “mark” can be an author of a book, but not a movie.

If you want to have a virtual author “hhhh” that can be associated only with books but not with movies, then it is not a real author and the best way to indicate that is to have a separate “virtual author” taxonomy.

Translating to your question…. sounds like you have categories which are not content categories, and therefor do not belong there in the first place, and should be a different taxonomy or meta data