In your original setup all category queries on the frontend were affected by the
pre_get_posts filter you defined. Hence its interference with the menu query. Then you added an extra condition,
is_main_query. The result being that both the menu and your main loop were no longer affected by the filter.
This can mean only one thing. What looks like the main query on your page actually is not. So the filter is not applied.
If you have written this theme yourself you could clean it up so you get the main query and the filter is applied. Otherwise, you could examine the code if there are some arguments set, which you could check with the
get method to make sure you are conditionally targeting the right query in stead of using