I’ve figured out the issue. Here is the working code:
foreach ( get_the_terms( $posts[0]->ID, 'colors' ) as $term ) {
$term_link = get_term_link( $term, 'colors' );
if ( false !== strpos( $referrer_url, $term_link ) )
return $term;
}