How to set custom editor style when editing the homepage?

If I understand you correctly, you want a custom editor style when editing the homepage? You can check if you’re currently editing the homepage by comparing the post ID to the ID in the page_on_front option, like this:

function homepage_editor_styles() {
    global $post_ID, $post_type;

    if ( empty ( $post_ID ) || 'page' !== $post_type )
        return;
    if ( $post_ID === (int) get_option( 'page_on_front' ) ) {
        add_editor_style( 'css/editor-style-homepage.css' );
    }
}
add_action('admin_head', 'homepage_editor_styles');