It looks like, in a Multisite installation, any network-wide plugins record their activation time in a site option. By default, plugins activated in a single site in a Multisite network (or in a single-site installation) do not.
See the source of the activate_plugin() function.
You could write a simple plugin that records that information, using the activated_plugin hook, but that won’t help you with any older plugins — any data you record that way would only exist from now[^1] on out.
[^1]: If by “now” we mean “the moment you activate your plugin-tracking plugin”.