How to correctly load wordpress in a non WP script for AJAX request

It is really not best practice to do it this way at all. In fact you don’t need to create an ajax.php file unless you’re just going to include it in your functions.php file. You need to read up on AJAX in WordPress. You would just add an action on ‘wp_ajax_name_of_action’ and just specify the action within the javascript ajax function.

Simple example from the WordPress codec:

<?php
add_action('admin_head', 'my_action_javascript');

function my_action_javascript() {
?>
<script type="text/javascript" >
jQuery(document).ready(function($) {

var data = {
    action: 'my_action',
    whatever: 1234
};

// since 2.8 ajaxurl is always defined in the admin header and points to admin-ajax.php
jQuery.post(ajaxurl, data, function(response) {
    alert('Got this from the server: ' + response);
});
});
</script>
<?php
}
add_action('wp_ajax_my_action', 'my_action_callback');

function my_action_callback() {
global $wpdb; // this is how you get access to the database

$whatever = intval( $_POST['whatever'] );

$whatever += 10;

    echo $whatever;

die(); // this is required to return a proper result
}