Unfortunately there is no filter which can overcome this issue. You have to override the start_el function of the walker. Here is the original source:
$class_names = join( ' ', apply_filters( 'nav_menu_css_class', array_filter( $classes ), $item, $args ) );
$class_names=" class="" . esc_attr( $class_names ) . '"';
$id = apply_filters( 'nav_menu_item_id', 'menu-item-'. $item->ID, $item, $args );
$id = strlen( $id ) ? ' id="' . esc_attr( $id ) . '"' : '';
$output .= $indent . '<li' . $id . $value . $class_names .'>';
As you can see, class is always appended, and it should be appended only if $classes is not empty