I want to allow certain file types on dokan upload files

You can try allowing mimes yourself, like this :

add_filter('upload_mimes', 'custom_upload_mimes');
function custom_upload_mimes($existing_mimes) {
    $existing_mimes['rbxlx'] = 'text/xml'; 
    $existing_mimes['rbxlm'] = 'text/xml'; 
    $existing_mimes['rbxl'] = 'text/xml'; 
    return $existing_mimes;
}

If this doesn’t help, you need to look outside WP – it’s possible the server itself won’t allow those uploads. Can you access the .htaccess ? If so, you can try adding this :

<IfModule mod_mime.c>
    AddType text/xml .rbxlx
    AddType text/xml .rbxlm
    AddType text/xml .rbxl
</IfModule>

If not, you can try contacting your hosting.