The “magic hybrid of jQuery and php” is called “AJAX”, at least in this case. You will need to use the Javascript to conditionally make another request to the server. The PHP runs on the server and so has no idea how tall the page is. The Javascript runs in the browser and can work out the window height but can’t run the PHP directly. The only choice is AJAX.
WordPress has an AJAX API, that makes AJAX requests very easy. There are plenty of examples in the Codex for using the AJAX API and plenty of questions here about it as well. Get started. If you have trouble, edit your question with the specifics.