It depends. If you have a caching plugin installed then a page that is not visited often might not be cached for a while, or the cache could expire between visits. In that case the next visit would require the cached page to be regenerated, which will be much slower than loaded the cached version. But any visits after that should be faster, at least until the cache expires (how long that will take depends on your settings).
