Can admin-ajax.php be used for spam purposes? And if yes, how to prevent that?

Question

This is why you use nonces.

$.ajax({
    type: "POST",
    url: '/wp-admin/admin-ajax.php',
    data: { action: 'mail_function', message: 'test', _nonce: <?php echo wp_create_nonce( 'mail_function_' . $post->ID ) ?>},
    dataType: "html",
    success: function(data) {
    }
});

Then in your PHP function:

function my_ajax_mailer() {
    if ( ! wp_verify_nonce( $_REQUEST['_nonce'], 'mail_function_' . $post->ID ) )
        return;
    // send mail...
}