How to use filter on comment submission/insert

There is no wp_notify_postauthor filter, a pluggable function is one you override with your own function:

if( ! function_exists('wp_notify_postauthor') ) {
    function wp_notify_postauthor( $comment_id, $comment_type="" ) {
        // your own wp_notify_postauthor code
    }
}

I suggest looking at the function in source to see what it does.