You can achieve this using the quicktag_settings
filter
function wpse41427_remove_quicktags( $qtInit ) {
$qtInit['buttons'] = 'strong,em,link,block,del,img,ul,ol,li,code,more,spell,close,fullscreen';
return $qtInit;
}
add_filter('quicktags_settings', 'wpse41427_remove_quicktags');
The default buttons for quicktags is a comma seperated value for the buttons key:
$qtInit['buttons'] = 'strong,em,link,block,del,img,ul,ol,li,code,more,spell,close,fullscreen';