Can you limit the memory usage of a particular plugin?

No and even if you could, if the plugin ran out of available memory then the entire page generation would stop due to the fatal error.

You’re better off fixing the plugin itself to not use as much memory or to just further increase the total memory allocated to WordPress/PHP.