Instead of contacting the remote site to get the information in an expensive HTTP request, why not have the remote site send you the data at regular intervals?
- Register a REST API endpoint to recieve the data on
- In that endpoint, take in the data and save it in
all.json
- In that endpoint, take in the data and save it in
- On the remote site add a cron job
- grab the data
- make a non-blocking remote request to the site with this data
Now we have a completely asynchronous non-blocking system. Why fetch the data when it can be given to you?
This gives us several bonuses:
- The
save_postfilter can be completely removed making post saving faster - This fixes a number of bugs in the filter by removing the filter
- Updates to the file can happen even when no posts are being created
- Updates to the file are now predictable and routine, e.g. every 5 minutes, or every hour
- This avoids race conditions where multiple requests are sent to the API at the same time, resulting in extra server load and broken JSON files
- Your API endpoint takes a little time to figure out the JSON data, so this gives you control over how often it happens, e.g. if the site is struggling change the cron job from 5 minutes too 10 minutes to ease the load
- You could ping the API and tell it to trigger sending the data to the endpoint when a post is saved, rather than doing the full fetch and save. This would allow you to use a fetch paradigm and still have the advantages. It’s similar to how some payment and authentication flows work too.