Workaround AJAX calling without directly calling WordPress backend (Admin dashboard) because it is restricted to the end user (the public)

I guess, that whitelisting the endpoint is not possible, as this is the simplest solution here.

It will be a little workaround here, but you can try to replace your AJAX endpoint, with a REST API endpoint.

Your main upload logic should stay the same, request will be similar. Maybe you can even use the default WordPress endpoint:
https://developer.wordpress.org/rest-api/reference/media/

If no, just register your own endpoint:
https://developer.wordpress.org/reference/functions/register_rest_route/