It’s not possible with just WordPress to schedule code to run in the background at a specific time. WordPress, like most (if not all) PHP web applications, only actually runs in response to a URL being accessed, and a request being made to the server. For a scheduled event to run at a certain time, you would need to have something running on the server, such as an actual cron job, which can run continuously and trigger scheduled events even when WordPress has not been run.
WordPress alone is not capable of modifying the server behaviour itself to put this in place, but WordPress needs to be able to schedule tasks out of the box, without requiring server expertise from users, or additional server configuration beyond the standard install. To do this it checks for scheduled events whenever it is run, and executes any events whose scheduled time has passed. For high traffic sites, this could be as soon as immediately, but for very low traffic sites this could mean that code is executed a day late, or more.
The most straightforward solution for this is to configure a real cron job on your server to load WordPress automatically as frequently as you need. The method for doing this depends entirely on what type of server/hosting you have, and access to make this kind of change. WordPress can’t know every type of hosting environment, and doesn’t have this access for security reasons, which is why it can’t do it itself. One method, for SiteGround, is described here, but you should ask your host for help doing it in your own environment.