How to update all posts but the current one (post__not_in not working?)

There’s a more elegant way of updating your posts using the global $wpdb.

function rsc_unpublish_all_ads( $post_id, $post, $update ) {

// select all ads other than the current one
$args = array(
    'nopaging' => true,
    'post__not_in' => array($post_id),
    'post_type' => 'rsc-ads',
    'post_status' => 'publish',
    'fields'=>'ids'
);

$query = new WP_Query($args);

$published_ads = $query->posts;//array of post ids

 if ( !empty($published_ads) ) {

    global $wpdb;

    $ids = implode(',', $published_ads);

    $wpdb->query("UPDATE $wpdb->posts SET post_status="draft" WHERE ID IN ($ids)");

 }

}
    add_action( 'save_post_rsc-ads', 'rsc_unpublish_all_ads', 10, 3 );

I think you were having a problem calling wp_update_post() while the function was still running. In this implementation, if the $query is returning the post IDs properly, you make one trip to the database and fix all the posts without running a loop. Note I changed your hook to be specific to the “rsc-ads” post type so it will only fire on this post type.