Generally, you need to tell WordPress when to do things. If you just dropped your code in a plugin, then you’re not giving WordPress information on when to execute your commands, and you risk the functions you need not being defined yet.
Right now, your code is saying, “If someone isn’t logged in, always redirect them to the regular login URL,” which is the opposite of what it sounds like you’re trying to do.
Try a function in your plugin file, and attaching it to a hook. Here’s an example:
function wpse_177404_login_screen_redirect()
{
global $pagenow;
if ($pagenow == 'wp-login.php' && !is_user_logged_in()) {
wp_redirect('http://your-site.com/custom-login/');
exit;
}
}
add_action('init', 'wpse_177404_login_screen_redirect');
With that, you’re telling WordPress to check if the page is wp-login.php on every load, and if someone isn’t logged in, redirect them to the new page of your choosing.
Be careful when you over-ride these types of Core functionality, as it also impacts password resetting and registration, and can be very easy to frustrate your users by doing these sorts of things.