Is there any background process that I can run from plugin without depending on page hits on a website without affecting page-load speed?

There is no reason for your design, but the core of your problem is that (I assume) people are trying to use a free tier of the API instead of properly paying for its use and removing the ridiculous restrictions.

Anyway, the solution is fairly simple, run your cron every half an hour, allocate to your import process unlimited time (and you might need to instruct your users to increase memory limit), and just loop until you got all the requested data with a second delay http://php.net/manual/en/function.sleep.php between each request.

Even the worst case of each request taking 5 seconds to complete, the whole thing should be done in 5 minutes.

As for the user view, you should keep new info in some kind of limbo state until you got all of it, and only then make it presentable.

Related Posts:

  1. How Do I Make WordPress Run an Event Every Day?
  2. Hourly WP schedule, do I need at least 1 visitor hourly?
  3. WordPress cron running twice
  4. Will cron job run if page loaded is being served from cache?
  5. How to schedule a cron job in plugin without waiting for page load request?
  6. How to Trigger WP CRON at Local Timestamp?
  7. Plugin with functions inside a class & how to trigger WP CRON
  8. WP Cron as Fast as WordPress AJAX?
  9. What is the use of wp_schedule_event hook?
  10. Running one-off cron jobs when WP_DISABLE_CRON is true: can I hit /wp-cron.php?
  11. How do I log plugin (cron) actions?
  12. How to add Request header in WordPress remote api calls
  13. Which action for triggering cron “wp”or “init”?
  14. wp_schedule_event in a class
  15. WP Cron – Run Cron after evey 15 minutes at a specific time for the next three hours
  16. Cron firing my function everytime wp-admin is visited
  17. Using wp_cron with custom time interval and the register activation hook
  18. WordPress scheduled task is called but not executed
  19. Problem creating cron job wordpress
  20. How can I change the frequency of a scheduled event?
  21. get_posts / WP_Query Memory size of 134217728 bytes exhausted
  22. WP Cron doesn’t save or in post body
  23. Scheduled event does not run at midnight
  24. Is it possible to load plugin from console with core ?
  25. How ( and mostly at what time ) can i prevent the alternate cron from running?
  26. wp_remote_get keeps timing out
  27. Does wp-cron runs all tasks scheduled at same time together or one after other?
  28. Scheduling a task using class methods
  29. What is minimum time interval for which a cron be scheduled?
  30. How to Download and install plugin database remotely
  31. On Plugin Activation, How Do I Check for Proper Transport Mechanism?
  32. Use wp_set_post_terms() instead of wp_insert_post()
  33. Programmatically detect if wp-cron is disabled?
  34. Loading WordPress Admin Environment
  35. How to perform a heavy and long process in cron jobs?
  36. How does task scheduler plugin implements cron that is not dependent on page load request? [closed]
  37. How wp-cron can run in background if PHP is single threaded?
  38. Strange Situation When Try To Retrieve Github Gist Using wp_remote_get
  39. WordPress daily cron is executing more frequently than once a day
  40. Problems with cron
  41. wp-cron and execution of code in is_admin() included script
  42. WP Cron being triggered but not working as expected
  43. wp_schedule_event() set daily, but processed every second
  44. Will I get an error if I try unscheduling a WP Cron scheduled task that wasn’t scheduled?
  45. Looking for a better way to initiate cron job
  46. How to Schedule Cronjobs for start of every month and year
  47. Switch from wp-cron to a server cron job
  48. WordPress plugin cron working only if admin is logged in
  49. How to make a cronjob type plugin
  50. AJAX request not routing through proxy
  51. wp_schedule_single_event is set correctly but sometimes not fired
  52. WordPress cron job running more than once
  53. Simulating timeout is not working
  54. Cron: Update four post at Hour
  55. Activate Plugin Automatically After Set Time
  56. How to Run Plugin on Server Without Web Browser
  57. Better way to run heavy scripts using WordPress database
  58. add pagination to wp_remote_get
  59. Custom recurrence not working / Wp Cron event
  60. How and when would I implement some code which would take data from one part of the Db and put it in another?
  61. WordPress Cron function is not working
  62. Detect each active instance of a widget
  63. Best place to keep files that are called by cron jobs?
  64. Adding custom cron_schedule prior to wp_schedule_event
  65. Can wp_schedule_single_event be used to run background proccess?
  66. Call do_action() within WordPress cronjob
  67. wp cron does not run when i am not logged in to wp admin
  68. How can a plugin create a page/form in the front end?
  69. Looking for WordPress System Diagrams
  70. Edit the output of wp_widget_rss_output()
  71. Does WP Have a Function To Generate .htaccess RewriteCond?
  72. Changing bloginfo description from a plugin
  73. wp_enqueue_style built in styles
  74. WordPress Plugin Setting’s POST
  75. wp_localize_script and host/browser cache
  76. Get a custom table to an array
  77. Error : “Cannot use object of type WP_Post as array in”
  78. Replacing Scripts in Admin Load_Scripts
  79. How to filter content on Save/Publish to add rel=”nofollow” to all external links?
  80. How to show active version on wordpress.org of a published plugin?
  81. Find out Requires WP tag for a plugin when submitting it
  82. Plugin Architecture Question: How to avoid using transients
  83. SSL certificate verify failed
  84. How to load library scripts in admin from plugins in noConflict wrapper?
  85. Additional classes `undefined` after saving post in Gutenberg
  86. How do you submit a Gutenberg plugin?
  87. WP nonce field checkbox prints checked=’checked’ outside input field
  88. Creating two tables in database on activation hook
  89. It is possible to pass $args that sent by add_settings_field() inside another function?
  90. Change reminder email date to 14 days before
  91. Adding Amchart Interface to WordPress API
  92. WordPress Own Rewrite Rules
  93. Don’t print customizer styles when no setting has been used
  94. WP_CRON issue with UTC and local time
  95. WordPress: Cron locking and Queue
  96. Can I access WordPress API’s from within plugin scripts?
  97. Is there a way to make is_search() always return false
  98. Apply Filters Causing a 500 Internal Server Error
  99. WP Job Manager – Show recent jobs as cards
  100. Admin Menu for New Post