Dequeue set-post-thumbnail.min.js

You should be able to dequeue like normal. Despite the complexity of the function, it is really just enqueueing scripts and styles. Tested with the admin bar script on the front end:

add_action(
  'wp_enqueue_scripts',
  function() {
    wp_dequeue_script('admin-bar'); 
  }
);

You just want admin_enqueue_scripts and set-post-thumbnail:

add_action(
  'admin_enqueue_scripts',
  function() {
    wp_dequeue_script('set-post-thumbnail'); 
  }
);

I am not sure of the wisdom of dequeueing a Core script though. Be warned.