I may be late to this party, but to stop plugin activation and have WordPress show an error message where the admin notices go, I simply output an error message and terminate execution. This has the added advantage of playing nice with wp-cli:
Example:
class testPlugin() {
...
static function activate() {
//[do some stuff here]
if ($error) {
die('Plugin NOT activated: ' . $error);
}
}
register_activation_hook( __FILE__, array( 'testPlugin', 'activate' ));