First of all you need prevent whole post form sending.
Try:
function rt_custom_button(){
$html="<div id="major-publishing-actions" style="overflow:hidden">";
$html .= '<div id="publishing-action">';
$html .= '<button onclick="send_note(event);" accesskey="p" tabindex="5" class="button-primary" id="custom" name="">send media note</button>';
$html .= '</div>';
$html .= '</div>';
echo $html;
}
add_action( 'post_submitbox_misc_actions', 'rt_custom_button' );
Then you need to add JS function somewhere:
function send_note(event){
event.preventDefault();
$.post(ajax_url, {
action: 'send_note'
}, function(response){
...
});
}