Inserting Hundreds of Thousands of Posts at Once

There are two reasons I can think of that may cause negative impact

1. wp_insert_post retrieves the post from the DB after writing it to it. this is not fun but should have only a constant impact, so I’m guessing it is not your main problem

2. Each post is being stored in memory as part of the caching process, and in the end this makes your server choke when it runs out of physical memory and starts swapping. try calling wp_suspend_cache_addition which suspends caching in the beginning of your script.

wp_suspend_cache_addition(true);

echo "get list of posts to be created in wordpress \n";