As @milo said in the comments, the cat parameter (and similar parameters in other hierarchical taxonomies) will make the query look in subcategories as well. If you do not want that to happen use the category__in parameter.
As for the general slowness, it is hard to guess, but it depends on your DB server performance (maybe it is under constant load and it is time to upgrade it) and how “deep” are those posts in the table.
Last but not least, if you do not need the information of how many matching results exist beyond your limit, you should try adding 'no_found_rows' => true to your query. That way the DB will not try to calculate the number of total posts in the category (the SQL_CALC_FOUND_ROWS part of the query)