Is there any background process that I can run from plugin without depending on page hits on a website without affecting page-load speed?

There is no reason for your design, but the core of your problem is that (I assume) people are trying to use a free tier of the API instead of properly paying for its use and removing the ridiculous restrictions.

Anyway, the solution is fairly simple, run your cron every half an hour, allocate to your import process unlimited time (and you might need to instruct your users to increase memory limit), and just loop until you got all the requested data with a second delay http://php.net/manual/en/function.sleep.php between each request.

Even the worst case of each request taking 5 seconds to complete, the whole thing should be done in 5 minutes.

As for the user view, you should keep new info in some kind of limbo state until you got all of it, and only then make it presentable.

Related Posts:

  1. How Do I Make WordPress Run an Event Every Day?
  2. Hourly WP schedule, do I need at least 1 visitor hourly?
  3. WordPress cron running twice
  4. Will cron job run if page loaded is being served from cache?
  5. How to schedule a cron job in plugin without waiting for page load request?
  6. How to Trigger WP CRON at Local Timestamp?
  7. Plugin with functions inside a class & how to trigger WP CRON
  8. WP Cron as Fast as WordPress AJAX?
  9. What is the use of wp_schedule_event hook?
  10. Running one-off cron jobs when WP_DISABLE_CRON is true: can I hit /wp-cron.php?
  11. How do I log plugin (cron) actions?
  12. How to add Request header in WordPress remote api calls
  13. Which action for triggering cron “wp”or “init”?
  14. wp_schedule_event in a class
  15. WP Cron – Run Cron after evey 15 minutes at a specific time for the next three hours
  16. Cron firing my function everytime wp-admin is visited
  17. Using wp_cron with custom time interval and the register activation hook
  18. WordPress scheduled task is called but not executed
  19. Problem creating cron job wordpress
  20. How can I change the frequency of a scheduled event?
  21. get_posts / WP_Query Memory size of 134217728 bytes exhausted
  22. WP Cron doesn’t save or in post body
  23. Scheduled event does not run at midnight
  24. Is it possible to load plugin from console with core ?
  25. How ( and mostly at what time ) can i prevent the alternate cron from running?
  26. wp_remote_get keeps timing out
  27. Does wp-cron runs all tasks scheduled at same time together or one after other?
  28. Scheduling a task using class methods
  29. What is minimum time interval for which a cron be scheduled?
  30. How to Download and install plugin database remotely
  31. On Plugin Activation, How Do I Check for Proper Transport Mechanism?
  32. Use wp_set_post_terms() instead of wp_insert_post()
  33. Programmatically detect if wp-cron is disabled?
  34. Loading WordPress Admin Environment
  35. How to perform a heavy and long process in cron jobs?
  36. How does task scheduler plugin implements cron that is not dependent on page load request? [closed]
  37. How wp-cron can run in background if PHP is single threaded?
  38. Strange Situation When Try To Retrieve Github Gist Using wp_remote_get
  39. WordPress daily cron is executing more frequently than once a day
  40. Problems with cron
  41. wp-cron and execution of code in is_admin() included script
  42. WP Cron being triggered but not working as expected
  43. wp_schedule_event() set daily, but processed every second
  44. Will I get an error if I try unscheduling a WP Cron scheduled task that wasn’t scheduled?
  45. Looking for a better way to initiate cron job
  46. How to Schedule Cronjobs for start of every month and year
  47. Switch from wp-cron to a server cron job
  48. WordPress plugin cron working only if admin is logged in
  49. How to make a cronjob type plugin
  50. AJAX request not routing through proxy
  51. wp_schedule_single_event is set correctly but sometimes not fired
  52. WordPress cron job running more than once
  53. Simulating timeout is not working
  54. Cron: Update four post at Hour
  55. Activate Plugin Automatically After Set Time
  56. How to Run Plugin on Server Without Web Browser
  57. Better way to run heavy scripts using WordPress database
  58. add pagination to wp_remote_get
  59. Custom recurrence not working / Wp Cron event
  60. How and when would I implement some code which would take data from one part of the Db and put it in another?
  61. WordPress Cron function is not working
  62. Detect each active instance of a widget
  63. Best place to keep files that are called by cron jobs?
  64. Adding custom cron_schedule prior to wp_schedule_event
  65. Can wp_schedule_single_event be used to run background proccess?
  66. Call do_action() within WordPress cronjob
  67. wp cron does not run when i am not logged in to wp admin
  68. Cron Job not working
  69. WordPress and PHP Sessions – Security and Performance
  70. Must-use plugins autoloader: How to use get_plugins() correctly?
  71. Changing upload directory for plugin uploads only
  72. Upload file to remote storage
  73. One time username change from frontend?
  74. How to not let a user with a new role edit users that have administrator role?
  75. async code or cron job?
  76. How to Better Control WordPress Cron Jobs?
  77. WordPress not working on localhost
  78. Custom delete option button in plugin settings
  79. Storing data in wordpress database from ajax call from different website
  80. Prevent invalid or empty values from being saved to the database and retain the form field values upon error
  81. Hide shortcodes “code” when using the “editor” the same as wordpress do with “gallery”
  82. wordpress JSONAPI introspector always limits number at 10?
  83. Custom Post Type to override theme’s CSS & HTML from Plugins Dir?
  84. Move some files of plugin to the root directory with plugin activation
  85. Unable to get a simple plugin admin page to work
  86. How do I get rid of my inclusion race-condition on wp_enqueue_script
  87. AJAX call to admin-ajax.php by subscriber returns home page
  88. Best way to ping for the API changes in the wordpress?
  89. WordPress plugin advice: suggestions for writing a plugin the correct way
  90. Creating an ics calendar from custom post type
  91. Catch metabox values in Plugin
  92. How to show the posts listing on dashboard or custom page in admin panel?
  93. How put the correct URL to sub-menu plugin?
  94. WordPress-child problem with many css files in parent theme
  95. Can a standalone app be made to work as a WordPress plugin?
  96. Create and style menu
  97. Get URL of Post You Are Editing
  98. Force remove parent theme CSS/JS and call them from Child theme
  99. Output HTML Tags In Gutenberg Block
  100. Can you nest columns/column in a gutenberg custom template?