It is hard to give a 100% answer without seeing the code of how the shortcode is being registered, but from your description it sounds like it is not being registered when ajax requests are being processed. The reason is probably because ajax requests are being handled in an “admin” conttext, and shortcodes are a front end artifact and therefor the plugin author might decide to not register at all when on admin side.
there are several possible solutions
- Do not (and try never to) use
do_shortcode
just call the api that generates the shortcode directly - Register the shortcode yourself.
- Use the
wp-json
endpoint with register_rest_route instead of the theadmin-ajax.php
one. It do not assume admin context and it is somewhat faster as it do not do admin related initialization.