You’ve touched on a few things here, and there’s both the server and the frontend response time that you’re battling with. You might have issues with both.
It’s worth noting firstly that PageSpeed (and the style recalcuating alert in the Chrome timeline) both deal with front-end aspects only – in slightly different ways though (another good tool to check out is GTmetrix).
As far as the backend goes, you could try the Plugin Profiler plugin to see which plugins are causing the most issues. Or alternatively – and probably a much better solution – disable every plugin, change your theme, then enable one by one to see where the bottleneck is.
If there’s a lot of style recalculations, it might be some bad coding on the part of your theme. If you can prove that the delay is due to your theme (by changing it to another one to see if the delay goes away), it might be worth writing to the theme authors to see if there’s anything they can do about it.
Another item worth checking is the media on the page. Have you got any super large images that you could optimise, that could be adding to the time the browser takes to download your page?
In the end, there are many, many things that could be causing delays both on the backend and the frontend. You’ve already confirmed you have a good enough host because your other site is working fine, so the problem is going to lie somewhere within what is unique to this site.
Good luck and feel free to post comments/add additional information to your post if there’s any aspect of this you have questions about!