Your calls to get_the_post_thumbnail do work, the problem is how what you’re doing with the result:
<input type="checkbox" name="<?php echo $this->get_field_name( 'photo' ); ?>"
value="<?php echo esc_html(get_the_post_thumbnail( $post->ID, 'thumbnail' ) ) ?>"
<?php checked( $instance['photo'], get_the_post_thumbnail( $post->ID, 'thumbnail' ) ); ?> >
You can’t put image HTML tags inside of another tags attributes, that’s not valid html. You’re also using an escaping function that strips out HTML esc_html.
This will never work, and makes little sense:
<input type="checkbox" name="test" value="<img src="https://wordpress.stackexchange.com/questions/218406/example.com/img.png">">
Instead consider something that generates valid markup, such as this:
<label><input type="checkbox" name="test"><img src="https://wordpress.stackexchange.com/questions/218406/example.com/img.png"></label>
or
<input type="checkbox" id="testid" name="test">
<label for="testid"><img src="https://wordpress.stackexchange.com/questions/218406/example.com/img.png"></label>
See here for how to create labels for checkboxes, and remember to use esc_attr to escape your value