Is having thousands of unique categories a concern?

I’ve worked with sites that have thousands of categories and over 20K post tags. This really doesn’t deal with WordPress capability per se as much as server capacity and capability.

Here’s what I would recommend:

  1. Install the W3TC Plugin
  2. Install memcached on your server
  3. Set W3TC to use object caching via memcached – you will need to define your server and memcached port
  4. Boost your server memory via php.ini

These are preliminary steps to creating a more enterprise environment for your WordPress installation. If server-side caching doesn’t meet the needs of your traffic, you might want to look at a third-party caching service as well.