This approach (one general css) will make your first-load times longer, but when you browse other pages, if the css is cached, everything will be faster.
If you make your stylesheet smaller by splitting them up.. you might have a ‘slightly faster’ first-page load but every other page that uses another css will still be slower than using one generic css for all pages.
You can minify them if you want as what @JMa have said, you can use caching plugins or even a CDN like cloudflare if you want to make your site faster..
Here is a tool to check your page load times for free:
http://tools.pingdom.com/fpt/ or http://www.webpagetest.org/