WordPress custom login form using Ajax

A few issues I can see but no guarantee it’ll entirely fix the problem(s) –
can you ensure WP_DEBUG and WP_DEBUG_LOG are defined as true in your wp-config.php – any errors during runtime should then appear in wp-content/debug.log.

1. You have ! isset( $_POST ) as a conditional to check the request, but this will always be false because the $_POST superglobal is always set – it’s just empty unless there is POST’ed data. Instead, use $_SERVER['REQUEST_METHOD'] !== 'POST'
2. I can’t see any need for the wp_setcookie call. The function is deprecated and is just a wrapper for wp_set_auth_cookie, which is called & handled by wp_signon anyway.
3. If you send a redirect server-side (i.e. via PHP), the AJAX client (your browser) will silently follow the redirect and then return the response of said redirect for your callback (in your case, the homepage HTML). You would be better to send back JSON & instruct your JavaScript to do the redirect client-side (e.g. window.location = responseData.redirectUrl)