fs.hdfs.impl.disable.cache caused SparkSQL very slow

This is a question related to this question: Hive/Hadoop intermittent failure: Unable to move source to destination

We found that we could avoid the problem of “Unable to move source … Filesystem closed” by setting fs.hdfs.impl.disable.cache to true

However, we also observed that the SparkSQL queries became very slow — queries that used to finish within a few seconds now take more than 30 to 40 seconds to finish (even when the query is very simple, like reading a tiny table).

Is this normal?

My understanding of fs.hdfs.impl.disable.cache being true means that FileSystem#get() would always createFileSystem() instead of returning a cached FileSystem. This setting prevents a FileSystem object from being shared by multiple clients and it really makes sense, because it would prevent, for example, two callers of FileSystem#get() from closing each other’s filesystem.

(For example, see this discussion )

This setting would slow things down, but probably not by so much.

From: hadoop-source-reading

/**
 * Returns the FileSystem for this URI's scheme and authority. The scheme of
 * the URI determines a configuration property name,
 * <tt>fs.<i>scheme</i>.class</tt> whose value names the FileSystem class.
 * The entire URI is passed to the FileSystem instance's initialize method.
 */
public static FileSystem get(URI uri, Configuration conf)
        throws IOException {
    String scheme = uri.getScheme();
    String authority = uri.getAuthority();

    if (scheme == null) { // no scheme: use default FS
        return get(conf);
    }

    if (authority == null) { // no authority
        URI defaultUri = getDefaultUri(conf);
        if (scheme.equals(defaultUri.getScheme()) // if scheme matches
                // default
                && defaultUri.getAuthority() != null) { // & default has
            // authority
            return get(defaultUri, conf); // return default
        }
    }

    String disableCacheName = String.format("fs.%s.impl.disable.cache",
            scheme);
    if (conf.getBoolean(disableCacheName, false)) {
        return createFileSystem(uri, conf);
    }

    return CACHE.get(uri, conf);
}

Would the slowness point to some other networking issues, such as resolving domain names? Any insights to this problem are welcome.