Here is an example of the difference, as a spark-shell session: First, some data – two lines of text: Now, map transforms an RDD of length N into another RDD of length N. For example, it maps from two lines into two line-lengths: But flatMap (loosely speaking) transforms an RDD of length N into a collection of N collections, then flattens … Read more
Is just the max value .The goal is to calculate OVERHEAD as a percentage of real executor memory, as used by RDDs and DataFrames controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers. The value of the spark.yarn.executor.memoryOverhead property is added to the … Read more
Reading the Spark documentation I found an easier solution. Since version 1.4 of spark there is a function drop(col) which can be used in pyspark on a dataframe. You can use it in two ways df.drop(‘age’) df.drop(df.age) Pyspark Documentation – Drop