Hook to change Logout url

There’s no need to regenerate the logout URL (the $logout_url part in your code) because the first parameter passed to your function is already the logout URL.

So basically, just rename that $force_reauth to $logout_url and remove the $logout_url = wp_nonce_url( ... );:

function my_custom_logout_url($logout_url, $redirect=null){ // rename the $force_reauth
    // And remove this:
    //$logout_url = wp_nonce_url(site_url('logout.php')."?action=logout", 'log-out' );

Secondly, your add_filter() call is missing the fourth parameter (which is the number of arguments passed to your function):

add_filter('logout_url', 'my_custom_logout_url', 10, 2); // like this
add_filter('logout_url', 'my_custom_logout_url');        // not this

UPDATE

If you actually did mean to set the logout URL to example.com/logout.php (note the logout.php), then your code is actually good (except the add_filter() thing above). But if your code is still showing wp-login.php, then it’s possible another code (maybe a plugin) is filtering/changing the logout URL. And in that case, you can change the callback priority to a greater number like 20:

// The third parameter is the callback priority.
add_filter('logout_url', 'my_custom_logout_url', 20, 2); // the priority is now 20