Transferring/Uploading Data from DB to WordPress

Since you are building the scraping functionality outside of WordPress, you will have to use a way that allows you to add contents to WordPress from the outside as well. I basically see three options:

  1. Update via the REST API – Using the REST API you can post updates to the WordPress contents. You can do this right from your Python scraper code, posting to the REST API directly, when there is a change in the data. For example, you can create
    a post
    or update a post. The REST API is capable of handling
  2. Read the external database from WordPress – Alternatively, you could write a script that periodically checks the database where the Python scraper stores its data, for new changes. You could for example write a WP-CLI script that runs every x minutes (using a cron). That WP-CLI script in turn reads the external database and uses native WordPress functions like wp_insert_post() and wp_update_post() to save the data to WordPress.
  3. Build functionality to live read external database – Depending on what kind of data you are working with, importing the data might not be suitable. You could also make a page template that reads the external database while it is requested. The data doesn’t have to be stored in WordPress in order to be shown in WordPress pages. In this case, you would consider the database to be an external source, like an API, from where you fetch data as you need it.

Implementations like this often get really specific, so these are just pointers of the routes you could go. Depending on the type of data, the volume of the data and the interval in which the data is updated, one of these routes should suit your use case best.