Too many posts cause slow load when paginating

Yes, use caching. But even with caching you should rethink what you are actually doing because the cache for page/2 should change every time you publish a new post which might make caching pointless depending on the amount of traffic to that page.
You should probably rethink what you are trying to achieve there, why 54 and not less? 54 images on one page is a lot and the page will be slow not only because of the HTML generation but also because fetching the images will take time.