Force Cron to run only on one server

I would create a constant in the wp_config.php defining which Server you are on. So far so good, not really magic.

The tricky part, as you mentioned is to get the cron running for sure, and not aborting if it’s accidentally the wrong server. Try the function like this:

function my_cron_callback() {
    if ( MY_SERVER_SETUP == "SERVER_A" ) {
        do_the_cron_action();
    } else {
        set_transient( 'do_the_cron_on_server_a', 'true' );
    }
}

Now, you have a transient set when the cron would run on the other server, but is not executed. The Function do_the_cron_action contains your script.

The last step is to hook a function to the init, checking if the server is ServerA and if the transient is set.

function check_if_a_should_execute() {
    if( MY_SERVER_SETUP == "SERVER_A" && get_transient( 'do_the_cron_on_server_a' == 'true' ) {
        do_the_cron_action();
        delete_transient( 'do_the_cron_on_server_a' );
    }
} 
add_action( 'init', 'check_if_a_should_execute' );

That should do the trick.