As you are concatenating the result to a href you must wrap the ifelse statement inside another set of parentheses so the result is calculated first and then returned:
<a href="'.(($cat_id == 5) ? esc_url($custom_link) : esc_url($category_link)).'">