Protect custom php file with login

This is impossible to answer definitively without a better description, but I am assuming that app.php is a completely distinct piece of code. In which case, WordPress can’t really manage access to it. You will need to incorporate it into WordPress. While this is not the only way to do it I would advise:

  1. Create a custom page template for the app.
  2. In that page template paste your app code or include it.
  3. Create a “Page” from wp_admin->Pages and select your template file as the “Template”
  4. To control access you can include if ( is_user_logged_in()) { logic directly in the template. Do this if you want to provide some kind of “Must be logged message”

    if ( is_user_logged_in()) {
        // your app code
    } else {
       echo "Please don't do that";
    }
    
  5. Or, hook to template_redirect. This will let you redirect to some other page

    function protect_my_app_wpse_90691() {
      if (is_page_template('template-name.php') && !is_user_logged_in()) {
        wp_safe_redirect(get_bloginfo('url').'/wp-login.php'); // or some other page that you prefer
      }
    }
    add_action('template_redirect','protect_my_app_wpse_90691');