I suspect that even though the site is using a server-side page caching plugin, some part of WordPress still loads, and that delays the server’s response time. Static files will always be very fast, because there’s very little processing to be done.
Recommend reading articles on web.dev regarding optimizing the main thread. Start with this one: https://web.dev/optimize-long-tasks/.
The main thread can only process one task at a time. When tasks stretch beyond a certain point—50 milliseconds to be exact—they’re classified as long tasks. If the user is attempting to interact with the page while a long task runs—or if an important rendering update needs to happen—the browser will be delayed in handling that work. This results in interaction or rendering latency.
- It is always better to use as few plugins as possible. Plugins come with a lot of options and features to serve as many people as possible, which can be problematic for performance, especially if the developers are apathetic or ignorant of writing performant code. That said, without a comprehensive site audit, it would be difficult to know if plugins are causing performance problems.