Typically page builder themes and plugins are compatible with caching. The reason they still tend to slow down websites is because they are very resource-intensive.
In WP there are two main resources – data from the database, and files. Because page builders can do almost anything you can imagine, they have to come with enough CSS and JavaScript (and even HTML markup) to power all those infinite possibilities. So that page builder is going to come with hundreds of kilobytes of accompanying CSS and JavaScript to be able to retrofit all these exciting layouts on HTML that’s not really optimized for it.
Complex themes come with similar overhead. They may have huge CSS files and complex JavaScript, and they too often suffer from database bloat – especially if the developers have just thrown all of their options into the wp_options database table and have them all autoloading. The larger your options table becomes, the slower your whole site will perform. Also, even “less complex” themes – with fewer options – can still be coded in a way that makes them slower. Generally the fewer files it contains, the more likely it is to be a less-resource-intensive theme. If you can find a theme that doesn’t have to make use of a CSS framework like Bootstrap or Foundation, that will strip off additional overhead – they make it easy to add things like nav bars and rows and columns, but at the end of the day, if you find a cleverly-coded theme that uses CSS Grid (and flexbox fallbacks) instead of a framework, that’s hundreds of kilobytes you’re saving in files alone.
Since page builders and complex themes have tons of files for every visitor to download, even if you have cached everything as much as humanly possible, the fact remains that they will force every visitor to download tons of resources – CSS and JS – whereas a simple theme will be much lighterweight. Imagine downloading an animated GIF meme instead of streaming a 5-minute Youtube video – the GIF is just inherently a lot smaller of a file, so it will download faster.
One other thing that comes into play with highly complex themes and page builders is the fact that not only do they have those core CSS and JS files that everyone downloads on every page, they also often have conditional CSS and JS, just for specific layouts. When you mention a theme that has hundreds of files, keep in mind that even when they are perfectly cached, a visitor will still have to download however many of those files applies to each page he visits. So, when you use a particular Page Template that’s an extra resource – plus that PHP file probably also has its own specific CSS and JS that only apply to that template. Those aren’t already cached on the visitor side – they’ve hit your homepage and maybe a post, and now they’re going to this new content with new CSS and JS their browser hasn’t encountered before – so they have to download those new resources anyway.
Loading speed is a pretty complex topic, as there are just so many different factors – we haven’t even talked about how fast your database server or file server respond, and your choice of hosting provider can drastically affect those times, which in turn affect every file or database query you make. To get started exploring which themes and plugin sets work best on your own particular server, I would recommend copying your site over onto a staging site – preferably on the same server as your live site – and then trying a few plugins such as query monitor plugins to view what database queries are being run and how long they take, some performance plugins that for example can combine all of your separate JavaScript files into a single file which can sometimes help boost speed, and some plugin checker type plugins that try to determine which plugins are the most resource-intensive. Don’t do this on your main site, as some of these plugins are resource-intensive themselves and can cause problems for your visitors. But by experimenting with all these – and then switching to a basic theme, for example, to see how much that changed performance – and then turning off one plugin at a time to see how that impacted your site – you can get a better feel for what’s bogging down your site and how you can code really nice custom experiences without all these extra resources.