So I actually solved this and forgot to write my solution.
So in my wordpress directory structure I have a categories directory. So I was putting my custom category template there. Apparently wordpress uses that categories directory for the built in categories or something and doesn’t check it for custom ones. I just added my custom-slug.php to the root of the theme and it works.
I dont really like it personally because it doesn’t seem to organize, but they may have their reasons for that.