You can disable the caching if you set update_post_meta_cache and update_post_term_cache to false while setting the WP_Query arguments.
When caching is disabled the data will be requested from the database only on demand.
You can check my post in here.
If you need more info just let me know.