The value of “spark.yarn.executor.memoryOverhead” setting?

spark.yarn.executor.memoryOverhead

Is just the max value .The goal is to calculate OVERHEAD as a percentage of real executor memory, as used by RDDs and DataFrames

--executor-memory/spark.executor.memory

controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers.

The value of the spark.yarn.executor.memoryOverhead property is added to the executor memory to determine the full memory request to YARN for each executor. It defaults to max(executorMemory * 0.10, with minimum of 384).

The executors will use a memory allocation based on the property of spark.executor.memoryplus an overhead defined by spark.yarn.executor.memoryOverhead.

Related Posts:

  1. How to delete columns in pyspark dataframe
  2. How does createOrReplaceTempView work in Spark?
  3. What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
  4. How to change dataframe column names in pyspark?
  5. How to join on multiple columns in Pyspark?
  6. How to join on multiple columns in Pyspark?
  7. What is the difference between map and flatMap and a good use case for each?
  8. How to convert rdd object to dataframe in spark
  9. How to kill a running Spark application?
  10. What is spark.driver.maxResultSize?
  11. Spark RDD to DataFrame python
  12. How to change dataframe column names in pyspark?
  13. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  14. scalac compile yields “object apache is not a member of package org”
  15. Spark – Error “A master URL must be set in your configuration” when submitting an app
  16. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  17. map vs mapValues in Spark
  18. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  19. Converting Pandas dataframe into Spark dataframe error
  20. fs.hdfs.impl.disable.cache caused SparkSQL very slow

Leave a Comment