To redirect back to your custom login form after login fails try this code:
// hook failed login
add_action('wp_login_failed', 'my_front_end_login_fail');
function my_front_end_login_fail($username){
// Get the reffering page, where did the post submission come from?
$referrer = add_query_arg('login', false, $_SERVER['HTTP_REFERER']);
// if there's a valid referrer, and it's not the default log-in screen
if(!empty($referrer) && !strstr($referrer,'wp-login') && !strstr($referrer,'wp-admin')){
// let's append some information (login=failed) to the URL for the theme to use
wp_redirect($referrer . '?login=failed');
exit;
}
}
You’ll have to check also for an empty password or login – when login form from frontend (your custom login form) is submitted empty, you’ll end up on standard wp-login.php again. To prevent this, append this code:
add_action( 'login_head', 'my_frontend_login_no_pass_no_username' );
function my_frontend_login_no_pass_no_username(){
$referrer = add_query_arg('login', false, $_SERVER['HTTP_REFERER']);
if ( (!isset($_REQUEST['user_login']) || ( isset( $_REQUEST['user_login'] ) && trim( $_REQUEST['user_login'] ) == '' ) ) || (!isset($_REQUEST['user_pass']) || ( isset( $_REQUEST['user_pass'] ) && trim( $_REQUEST['user_pass'] ) == '' ) ) ){
wp_redirect( add_query_arg('login', 'failed', $referrer) );
exit;
}
}
If you’re interested in a proper login after registration fails too, check my blogpost on this topic.