Got it to work, finally!
Nothing like a good sleep to restart your brain 🙂
Updated code that works:
$item_output = $args->before;
if( $item->object == 'category' ) {
$category = get_category( $item->object_id );
$category_color = get_field( 'category_color', $category);
$item_output .= '<a style="color:' . $category_color . ';"'. $attributes .'>';
} else {
$item_output .= '<a'. $attributes .'>';
}
$item_output .= $args->link_before . apply_filters( 'the_title', $item->title, $item->ID ) . $args->link_after;
$item_output .= ( $args->has_children ) ? '<i class="fa fa-angle-down"></i></a>' : '</a>';
$item_output .= $args->after;