I’d say this is your key clue:
We noticed that the more pages we have the slower that open edit page
takes.
Have you evaluated the plugins and custom scripts you are using? I’d start with deactivating all plugins, and if the issue clears up, reactivate one by one to isolate where the problem might be.