I don’t think that the huge number of categories is responsible for slowing down your site. There might be other problems, like render blocking resources and loops.
I recommend you to run your WordPress site through Page Speed Insights. It will help you find the exact reasons behind the slow speed. It will also recommend you the way of solving the specific issues.