How to retrieve custom meta term of category taxonomy from WP Rest API?

First you need to tell WordPress to show this custom field in the API response:
register_term_meta('category', '_category_color', ['show_in_rest' => true]);

Then you can filter the category term API endpoint like so:

add_filter( 'rest_prepare_category',function($response, $item, $request){
    $color = get_term_meta( $item->term_id, '_category_color', true );
    $response->data['color'] = $color ?: '';
    return $response;
}, 10, 3);