How to stop the wp_mail function?

Filter 'phpmailer_init', not 'wp_mail'. To prevent sending the mail out reset the PHPMailer object.

Prototype (not tested):

add_action( 'phpmailer_init', 'wpse_53612_conditional_mail_stop' );

function wpse_53612_conditional_mail_stop( $phpmailer )
{
    ! my_condition() and $phpmailer = new stdClass;
}