Ways to load admin-ajax faster without initializing all plugins?

If your requests take 3-5 sec to finish, then there is most likely something else going on. The admin Ajax itself won’t take that much to load. You might be having some heavy plugin, or having some hooks running expensive meta queries on load.

You can try the REST-API, which is slightly faster than Admin-Ajax. But the response will be in JSON, instead of plain HTML, you have to note that one.

Also, there might be an issue with the TTFB. Try disabling all the plugins, revert to default theme (move your code to default theme) and check if the problem still persists.