Check IF single OR multiple custom data has been entered

Since you already using get_post_custom a slightly cleaner yet still dirty (php) way of doing it.

$sermondata = get_post_custom($post->ID);

if( isset($sermondata['sermon_audio']) || isset($sermondata['sermon_video']) || isset($sermondata['sermon_document']))
{ ?>
<div id="downloads">
// rest of your echo output
</div>
<?php } ?>

ps . Just make sure the value is not null, to be safe you can combine the check with array_key_exists() (makes it slower though).