At my firm, I see these two commands used frequently, and I’d like to be aware of the differences, because their functionality seems the same to me: 1 2 They both copy the data from the directory on HDFS into the directory for the table on HIVE. Are there differences that one should be aware … Read more
This is a question related to this question: Hive/Hadoop intermittent failure: Unable to move source to destination We found that we could avoid the problem of “Unable to move source … Filesystem closed” by setting fs.hdfs.impl.disable.cache to true However, we also observed that the SparkSQL queries became very slow — queries that used to finish … Read more
not able to create anything in hdfs I did But showing what is the problem? Solution: http://unmeshasreeveni.blogspot.com/2014/04/name-node-is-in-safe-mode-how-to-leave.html?m=1
It’s confusing, but hadoop.tmp.dir is used as the base for temporary directories locally, and also in HDFS. The document isn’t great, but mapred.system.dir is set by default to “${hadoop.tmp.dir}/mapred/system”, and this defines the Path on the HDFS where where the Map/Reduce framework stores system files. If you want these to not be tied together, you can edit your mapred-site.xml such that the definition of mapred.system.dir … Read more