How to get_term_meta on single custom post?

You will need to use get_the_terms() to get the taxonomy terms associated with the post and loop through them until you find one that has an image set:

$post.     = get_queried_object();
$terms.    = get_the_terms( $post, 'taxonomy_name_here' );
$image_url = null;

if ( $terms && ! is_wp_error( $terms ) ) {
    foreach ( $terms as $term ) {
        $attachment_id = get_term_meta( $term->term_id, 'category-image', true );

        if ( $attachment_id ) {
             $image_url = wp_get_attachment_image_url( $attachment_id, 'full' );
             break;
        }
    }
}

if ( $image_url ) {
    // Output image as needed.
}