The paging is very slow under a large number of articles

The two most obvious things you can do to reduce the load time of this query are the IN statement and the offset in your LIMIT statement.

The larger the number of items you’re checking for in the IN statement, the less efficient the query. See: https://stackoverflow.com/questions/4514697/mysql-in-operator-performance-on-large-number-of-values

The reason the offset is causing performance issues is the inefficiency of mysql’s ability to count. See: https://stackoverflow.com/questions/4481388/why-does-mysql-higher-limit-offset-slow-the-query-down

If you can, I recommend warming the cache after a new article is published. This resets the cache, rebuilding it via a system process rather than by user request. See: https://stackoverflow.com/questions/434259/what-is-a-warm-up-cache

If you rely on user requests to create the cache, it can lead to users requesting the same result multiple times before the cache is able to be built completely which is not a great position to be in.

If you can get something like varnish in front of your site, that will help a ton. Varnish prevents most requests from hitting php and it’s lightning-fast at serving requests.