Associate custom categories to different user IDs

For your situation, ideal solution would be this plugin: Restrict Categories

It allows you to restrict the users to add, view and edit categories.

Note: But before installing, make sure you have a backup. The most recent review had a problem of categories getting disappeared.