Define add_editor_style to specific post types?

The first issue is that add_editor_style() is usually called in Theme setup functions that get hooked into after_setup_theme – which is well before the query is set up and the post type determined. So, you’ll need to move add_editor_style() to a separate callback, that gets called after the query is setup, but before TinyMCE is initialized. Perhaps tiny_mce_before_init?

<?php
function wpse59547_add_editor_style() {
    global $post;
    $post_type = get_post_type( $post->ID );
    $editor_style="editor-style-" . $post_type . '.css';
    add_editor_style( $editor_style );
}
add_action( 'tiny_mce_before_init', 'wpse59547_add_editor_style' );

Then, you just need to create editor-style-post.css, editor-style-book.css, etc.

If you want to default to editor-style.css, use get_post_types( array( 'public' => true, '_builtin' => false ) ) to return an array of custom post types, and use that, e.g.:

$custom_post_types = get_post_types( array( 'public' => true, '_builtin' => false ) );
$editor_style = ( in_array( $post_type, $custom_post_types ) ? 'editor-style-' . $post_type . '.css' : 'editor-style.css' );