How to check if user is uploading/setting an image as a featured image?

Determining if an attachment actually is a featured image is easy:

get_post_thumbnail_id( get_post() );

which is a convenience function around

get_post_meta( get_post()->ID, '_thumbnail_id', true );

It is a bit more complicated if you know nothing than the File name or Url.

add_filter( 'wp_handle_upload_prefilter', function( $file ) {

    $url = wp_get_attachment_image_src( 
        get_post_thumbnail_id( get_post() ),
        'post-thumbnail'
    );

    // Output possible errors 
    if ( false !== strpos( $url, $file ) ) {
        $file['error'] = 'Sorry, but we only allow featured images';
    }

    return $file;
} );

As you can see, I do only compare the current file string with a posts featured image URl. Note that I do not know if this will work, it is not tested and the wp_handle_upload_prefilter might be too early.

Another option might be to use the last filter inside _wp_handle_upload():

apply_filters( 'wp_handle_upload', array(
    'file' => $new_file,
    'url'  => $url,
    'type' => $type
), 'wp_handle_sideload' === $action ? 'sideload' : 'upload' );

which handles sideload as well.

In theory I assume it’s too early to check if the featured image is actually the featured image before it is set. You might be able to revert the upload once it finished (delete file, reset post meta data), but that is far from elegant.

The meta box itself is registered with add_meta_box() and the box callback is post_thumbnail_meta_box, which calls _wp_post_thumbnail_html() to render the contents. When you look in there, you will see that there is actually a hidden field:

$content .= '<input type="hidden" id="_thumbnail_id" name="_thumbnail_id" value="' . esc_attr( $thumbnail_id ? $thumbnail_id : '-1' ) . '" />';

Now the name is _thumbnail_id, which you should be able to fetch with filter_input() (good) or with $_POST['_thumbnail_id'] (bad) inside a callback like wp_handle_upload_prefilter to determine if it actually was a featured image:

$featID = filter_input( INPUT_POST, '_thumbnail_id', … filters … );