adding custom classes for tables

In your case you need to add custom hook for the_content filter:

add_filter( 'the_content', 'wpse8170_add_custom_table_class' );
function wpse8170_add_custom_table_class( $content ) {
    return str_replace( '<table>', '<table class="mycustom-class">', $content );
}

Add this snippet to your functions.php file.