If you’d use the get_categories() function to get all the categories, you can then loop through them. Partial pseudo code, but this will do the trick:
$categories = get_categories( array() );
$colors = array();
foreach ( $categories as $category ) {
$cat_meta = get_option( 'meta_field_' . $category->slug );
if (isset($cat_meta['color'])){
$colors[ $category->term_id ] = $cat_meta['color'];
}
}
PS. I don’t know if the field in the options table is using the slug or the category id. The colors array will be filled with id’s of the category as key and value of the meta field as value.
PPS. get_categories() function call might need some extra parameters in the array, adjust to your liking using the available parameters.
PPPS. While my answer technically answers this question, this answer by s_ha_dum is actually a better approach (might require some more refactoring though).