Site only for users authenticated by different PHP application

Quick and dirty: add something that runs during the WordPress init action that checks your the domain.com login status, and dies if the person isn’t logged in. At a very basic level (if you don’t need a pretty “access denied” page or anything), you can do something as simple as this in a plugin:

function wpsx_26176_check_login() {

    // some logic to validate your cookies. here I'm just checking if a cookie exists
    // but obviously your validation would be more extensive
    if(!isset($_COOKIE['yourcookie'])) {
        wp_die('get out of here'); // If cookie is missing, just die
    }   

}
add_action('init', 'wpsx_26176_check_login');

What you’ll need to keep in mind if rolling out WordPress on a subdomain is that cookies are by default only applicable to the domain they’re set on. You’ll have to set the cookies written by your custom application to specifically be available across all your subdomains.