How to apply a function to the value $email of get_user_by to override email_exists?

Based on your code, I think the problem may have been that you were checking a variable that was undefined. You were checking $db_email, which in your function was an undefined value, so you’d be looking for an empty result. You need to check the value passed to email_exists() (which would be the email being validated in the form).

If the patterns applied in your pluggable get_user_by() function result in the equivalent of how you are obfuscating the emails, then you should get a correct match if the email exists.

function email_exists( $email ) {
    // You had this as checking $db_email, which would have been
    // undefined. It should be checking for the value passed to
    // this function - $email.
    $user = get_user_by( 'email', $email );
    if ( $user ) {
        $user_id = $user->ID;
    } else {
        $user_id = false;
    }
    return apply_filters( 'email_exists', $user_id, $email );
}