There is no reason for your design, but the core of your problem is that (I assume) people are trying to use a free tier of the API instead of properly paying for its use and removing the ridiculous restrictions.
Anyway, the solution is fairly simple, run your cron every half an hour, allocate to your import process unlimited time (and you might need to instruct your users to increase memory limit), and just loop until you got all the requested data with a second delay http://php.net/manual/en/function.sleep.php between each request.
Even the worst case of each request taking 5 seconds to complete, the whole thing should be done in 5 minutes.
As for the user view, you should keep new info in some kind of limbo state until you got all of it, and only then make it presentable.
Related Posts:
- How Do I Make WordPress Run an Event Every Day?
- Hourly WP schedule, do I need at least 1 visitor hourly?
- WordPress cron running twice
- Will cron job run if page loaded is being served from cache?
- How to schedule a cron job in plugin without waiting for page load request?
- How to Trigger WP CRON at Local Timestamp?
- Plugin with functions inside a class & how to trigger WP CRON
- WP Cron as Fast as WordPress AJAX?
- What is the use of wp_schedule_event hook?
- Running one-off cron jobs when WP_DISABLE_CRON is true: can I hit /wp-cron.php?
- WordPress Cron Job not importing products
- How do I log plugin (cron) actions?
- How to add Request header in WordPress remote api calls
- Which action for triggering cron “wp”or “init”?
- wp_schedule_event in a class
- WP Cron – Run Cron after evey 15 minutes at a specific time for the next three hours
- Cron firing my function everytime wp-admin is visited
- Using wp_cron with custom time interval and the register activation hook
- WordPress scheduled task is called but not executed
- Problem creating cron job wordpress
- How can I change the frequency of a scheduled event?
- get_posts / WP_Query Memory size of 134217728 bytes exhausted
- WP Cron doesn’t save or in post body
- Scheduled event does not run at midnight
- Is it possible to load plugin from console with core ?
- How ( and mostly at what time ) can i prevent the alternate cron from running?
- wp_remote_get keeps timing out
- Does wp-cron runs all tasks scheduled at same time together or one after other?
- Scheduling a task using class methods
- What is minimum time interval for which a cron be scheduled?
- How to Download and install plugin database remotely
- On Plugin Activation, How Do I Check for Proper Transport Mechanism?
- Use wp_set_post_terms() instead of wp_insert_post()
- Programmatically detect if wp-cron is disabled?
- Loading WordPress Admin Environment
- How to perform a heavy and long process in cron jobs?
- How does task scheduler plugin implements cron that is not dependent on page load request? [closed]
- How wp-cron can run in background if PHP is single threaded?
- Strange Situation When Try To Retrieve Github Gist Using wp_remote_get
- WordPress daily cron is executing more frequently than once a day
- Problems with cron
- wp-cron and execution of code in is_admin() included script
- WP Cron being triggered but not working as expected
- wp_schedule_event() set daily, but processed every second
- Will I get an error if I try unscheduling a WP Cron scheduled task that wasn’t scheduled?
- Looking for a better way to initiate cron job
- How to Schedule Cronjobs for start of every month and year
- Switch from wp-cron to a server cron job
- WordPress plugin cron working only if admin is logged in
- How to make a cronjob type plugin
- AJAX request not routing through proxy
- wp_schedule_single_event is set correctly but sometimes not fired
- WordPress cron job running more than once
- Simulating timeout is not working
- Cron: Update four post at Hour
- Activate Plugin Automatically After Set Time
- How to Run Plugin on Server Without Web Browser
- Better way to run heavy scripts using WordPress database
- add pagination to wp_remote_get
- Custom recurrence not working / Wp Cron event
- How and when would I implement some code which would take data from one part of the Db and put it in another?
- WordPress Cron function is not working
- Detect each active instance of a widget
- Best place to keep files that are called by cron jobs?
- Adding custom cron_schedule prior to wp_schedule_event
- Can wp_schedule_single_event be used to run background proccess?
- Call do_action() within WordPress cronjob
- wp cron does not run when i am not logged in to wp admin
- Cron Job not working
- Weekly cron is either missing the schedule or triggered too many times on Fridays
- Nested Hooks with do_action for performance reasons – safe/necessary?
- WordPress custom url without using any of the custom tags given
- Counter not working correctly
- Generate Static Page to Show Search Results/Detail for API
- User meta query using Wildcard
- How do I use a plugin to swap out the template file for a custom post type?
- WordPress with React: Saving and Using Data Collected with fetch
- use a (Polymer) web component within a plugin (or theme)?
- Custom Post not Deleting on Uninstall
- Scripts/styles not loading on cloned WP Site when logged in
- how to add security questions on wp-registration page and validate it
- Multiple instances of $opt_name in redux framework
- Don’t print customizer styles when no setting has been used
- Initializing rest endpoint issue
- Using AJAX to submit and return data inside the WordPress Plugin Boiler Plate framework
- Append php file to footer
- Integrate separate web app inside wordpress [closed]
- How to avoid conflict if a plugin already have included open library?
- Can’t get wpColorPicker to work in a widget
- WordPress OOP get current user question?
- Plugin Activation Causes wp_register errors
- Load custom translation in custom plugin fails
- Add quick edit functionnality to plugin table
- How to filter an action inside class in MemberPress plugin?
- AJAX call of function containing javascript which is not loaded (Plugin development)
- XMLHttpRequest to open PHP file responds with Missing Page
- Including content into an add_shortcode() function
- Get the current post/page URL with plain permalinks
- Hide / show settings field based on other field’s value
- How to integrate plugins into block editor