How can I control which plugins run in the dashboard?

Lots of plugins utilize is_admin to “disable” themselves when the backend is active, which is quite nice.

On the other hand I would not really fumble around in the backend. If something goes wrong you might lock yourself out, and if it´s only for a moment, I wouldn´t like that.

Furthermore I think you have some serious problem going on if your plugins affect the loading time of your backend so bad. Maybe you should check that first.