The link markup passes through the filter named, guess what, cancel_comment_reply_link.
Something along the lines of (not tested):
add_filter( 'cancel_comment_reply_link', function( $formatted_link ) {
return 'before' . $formatted_link . 'after';
} );
It will still get wrapped into <small> in comment_form() later though, which might or might not be relevant and unfortunately seems hard to get rid of.