Filter 'phpmailer_init'
, not 'wp_mail'
. To prevent sending the mail out reset the PHPMailer object.
Prototype (not tested):
add_action( 'phpmailer_init', 'wpse_53612_conditional_mail_stop' );
function wpse_53612_conditional_mail_stop( $phpmailer )
{
! my_condition() and $phpmailer = new stdClass;
}