I haven’t tried it, but this might do it:
global $wp_scripts;
if (is_a($wp_scripts, 'WP_Scripts')) {
$wp_scripts->queue = array();
}
Basically just resetting the scripts queue to blank. Should work, I think. You’d want this right at the top of your attachment template, probably.
From an optimization perspective, it would be faster to use the is_attachment() method to not enqueue the ones you want at all instead.