I keep the default themes in place, and keep them updated.
The WordPress Foundation will keep those themes updated with any security issues, so as long as you keep them updated on your site, I’m not concerned with security problems.
The advantage that you have by keeping them is for testing. When you are troubleshooting an issue, it is very nice to be able to switch temporarily to Twenty Twelve and confirm or eliminate your theme code as the cause of the problem.