Ajax endpoint is known to be relatively slow, but that primarily comes from loading WordPress core.
If you are seeing drastic difference between two endpoints (native Ajax vs yours custom) that both perform core load, then something is doing something that degrades Ajax endpoint on top of it.
I would profile Ajax endpoint to have better idea of what is actually taking that much time. Without that it’s impossible to make good guess for situation described.