Since you’re filtering with the rest_prepare_{post_type} filter, you could restrict it to the WP_REST_Posts_Controller::get_items() callback, with the rest_{post_type}_query filter:
add_filter( 'rest_post_query', function( $args )
{
add_filter( 'rest_prepare_post', 'api_remove_extra_data', 12, 3 );
return $args;
} );
where the post type is post.
Note that in general we always want to return the filter value and I don’t follow the logic your api_remove_extra_data() callback. Maybe it’s a relic from version 1.x?
This is how the filter is now defined in version 2:
/**
* Filter the post data for a response.
*
* The dynamic portion of the hook name, $this->post_type, refers to post_type of the post
* being prepared for the response.
*
* @param WP_REST_Response $response The response object.
* @param WP_Post $post Post object.
* @param WP_REST_Request $request Request object.
*/
return apply_filters( "rest_prepare_{$this->post_type}", $response, $post, $request );