My website is slow on the initial load

Maybe your PHP is running per CGI or FastCGI in a separate process that shuts down after a while of inactivity. Does it change if you register at an uptime monitoring service?
A cache plugin like W3 Total Cache may help too.