That is a huge question and it is going to be hard to answer. And it is going to take a lot of research and trail and error on your part.
As far as WordPress specific things go, analyze your code base. Poorly written plugins or themes can cause huge delays. A 3 second backend processing time is not typical for a stock WordPress so disable your plugins one by one and see if you see a difference. If you do, maybe that plugin can be re-written to be more efficient. Same with the theme.
Other options are not WordPress specific– server specs, server config, database config, etc. I suspect it is probably server related. My instinct is to say “overloaded server”, especially since your are seeing this delay even with caching. Cached pages should have very, very little backend processing.