Resource Issues: Importing external data into custom post type and keeping records up to date

One big help I have found with importing / updating large qty’s of posts is to use InnoDB tables and transactions, in batches of 200-500 posts (with more postmeta, use smaller post batches). It greatly improves performance, given that much of it is individual MySQL statements each of which must hit the db transaction log at some point. So:

  • issue a $wpdb->query('start transaction');
  • after (batch count), issue a $wpdb->query('commit'); and another $wpdb->query('start transaction');
  • on completion, issue a $wpdb->query('commit');
  • on failure / exception, issue a $wpdb->query('rollback');

After doing that, then look at your #2 for the postmeta, building a single SQL statement for inserting (or updating) all postmeta for a post at once, because that will also make a big difference once you have transactions.

Leave a Comment