Only append custom classes to nav menu items

Yes, they’re stored as metadata '_menu_item_classes' for each menu item so eg (updated to use separate function)

function custom_wp_nav_menu_css_class( $classes, $item, $args, $depth ) {
    $whitelist = array(
        //List of allowed menu classes
        'current_page_item',
        'current_page_parent',
        'current_page_ancestor',
        'first',
        'last',
        'vertical',
        'horizontal'
    );
    // Note array containing empty entry always created for menu item meta so filter out.
    if ( $optional = array_filter( get_post_meta( $item->ID, '_menu_item_classes', true ) ) ) {
        $whitelist = array_merge( $whitelist, $optional );
    }
    return array_intersect( $classes, $whitelist );
}
add_filter( 'nav_menu_css_class', 'custom_wp_nav_menu_css_class', 10, 4 );