Modify category listing API response

If you are using the API to respond with one or more categories, you are querying for terms and not for taxonomies. So you should not be using WP_REST_Taxonomies_Controller but rather WP_REST_Terms_Controller.

Instead of hooking to the filter rest_prepare_taxonomy, try hooking to rest_prepare_{$this->taxonomy} instead. As you are working with the taxonomy “category” use it as rest_prepare_category:

add_filter( 'rest_prepare_category', 'modify_api_taxonomy_response', 10, 3 );