Downgrade user from role that allows post creation

You can add actions to the set_user_role hook:

add_action( 'set_user_role', 'wpse98904_remove_demoted_user_posts', 10, 2 );
function wpse98904_remove_demoted_user_posts( $demoted_author_id, $role ) {
     if( 'subscriber' == $role ) {
         // In here you'd search for all the posts by the user
         $args = array(
                'numberposts' => -1,
                'author' => $demoted_author_id,
                'post_type' => '{your custom post type name}',
                // So if your custom post type is named 'xyz_book', then use:
                // 'post_type' => 'xyz_book',
         );
         $demoted_author_posts = get_posts( $args );
         foreach( $demoted_author_posts as $p ) {
             // delete the post (actually Trash it)
             wp_delete_post( $p->ID );
             // To force the deletion (ie, bypass the Trash):
             // wp_delete_post( $p->ID, true );
         }
     }
}

Reference

set_user_role hook — Codex

set_user_role hook in WP Trac

wp_delete_post() — Codex