Since you already using get_post_custom
a slightly cleaner yet still dirty (php) way of doing it.
$sermondata = get_post_custom($post->ID);
if( isset($sermondata['sermon_audio']) || isset($sermondata['sermon_video']) || isset($sermondata['sermon_document']))
{ ?>
<div id="downloads">
// rest of your echo output
</div>
<?php } ?>
ps . Just make sure the value is not null
, to be safe you can combine the check with array_key_exists()
(makes it slower though).