How to add custom color fields to the category edit page?

To add custom fields and save/update meta data to terms, you basically need to use these action hooks.

custom fields html for new term
{$taxonomy}_add_form_fields

save custom field data for new term, get data from $_POST
created_{$taxonomy}

custom fields html for existing term
{$taxonomy}_edit_form_fields

save custom field data for existing term, get data from $_POST
edited_{$taxonomy}

Save / update term meta
add_term_meta()
update_term_meta()

Get term meta
get_term_meta()

You can see the general idea how to use these hooks here, https://wordpress.stackexchange.com/a/296767/144392

Use saved meta data

One way to use the save term meta data is to get it directly on the category template and add it as inline style to an element.

// category.php
$color = sanitize_hex_color(get_term_meta(get_queried_object_id(), 'color', true));
<h1<?php echo $color ? ' style="color: ' . $color . ';"' : ''; ?>>Category title</h1>

Another way is to add the meta data as inline style with wp_add_inline_style() on wp_enqueue_scripts action.

// functions.php
function wpdocs_styles_method() {
    if ( is_category() ) {
        $id            = get_queried_object_id();
        $title         = sanitize_hex_color(get_term_meta($id, 'title_color', true));
        $bg            = sanitize_hex_color(get_term_meta($id, 'bg_color', true));
        $custom_css="";

        if ( $title ) {
            $custom_css .= ".title {color: {$color};}";
        }
        if ( $bg ) {
            $custom_css .= " .background {background-color: {$bg};}";
        }
        if ( $custom_css ) {
            // update the slug
            wp_add_inline_style( 'your-theme-styles-slug', trim($custom_css) );         
        }
    }    
}
add_action( 'wp_enqueue_scripts', 'wpdocs_styles_method' );

Color picker

You can use wpColorPicker script to turn text input fields into nice color pickers. Usage examples https://make.wordpress.org/core/2012/11/30/new-color-picker-in-wp-3-5/ and Color picker for posts and pages