Difference between hive.tez.container.size and tez.task.resource.memory.mb

hive.tez.container.size

This property specifies tez container size. Usually value of this property should be the same as or a small multiple (1 or 2 times that) of YARN container size yarn.scheduler.minimum-allocation-mb and should not exceed value of yarn.scheduler.maximum-allocation-mb.

As a general rule don’t put value higher than memory per processor as you want 1 processor per container and you want to spun up multiple containers.

You can find very detailed answer and a great architecture diagram on Hortonworks community answer here

tez.task.resource.memory.mb

Amount of memory used by launched task in TEZ container. Usually this value is being set in DAG but if it’s not then by default it will get set to 1024 MB.

All task in the DAG will have same size, it is helpful for container reuse and good for performance too.

Related Posts:

  1. Why is Fetch task in Hive works faster than Map-only task?
  2. Hive: how to show all partitions of a table?
  3. fs.hdfs.impl.disable.cache caused SparkSQL very slow
  4. Hive dynamic partitioning
  5. how to write subquery and use “In” Clause in Hive
  6. get “ERROR: Can’t get master address from ZooKeeper; znode data == null” when using Hbase shell
  7. Failed to locate the winutils binary in the hadoop binary path
  8. Name node is in safe mode. Not able to leave
  9. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
  10. First hadoop project error: “Input path does not exist”
  11. Difference between `load data inpath ` and `location` in hive?
  12. What is Keyword Context in Hadoop programming world?
  13. Converting bytes to megabytes
  14. How do I determine the size of my array in C?
  15. What does pss mean in /proc/pid/smaps
  16. Docker error : no space left on device
  17. Stack Memory vs Heap Memory
  18. How can I explicitly free memory in Python?
  19. How to clear variables in ipython?
  20. How to solve munmap_chunk(): invalid pointer error in C++
  21. How to clear variables in ipython?
  22. How does the “view” method work in PyTorch?
  23. Hadoop “Unable to load native-hadoop library for your platform” warning
  24. Python subprocess.Popen “OSError: [Errno 12] Cannot allocate memory”
  25. What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
  26. What does ** do in C language?
  27. Using Dynamic Memory allocation for arrays
  28. memory error in python
  29. In Python, what is `sys.maxsize`?
  30. How do I profile memory usage in Python?
  31. Difference between word addressable and byte addressable
  32. Python/Numpy MemoryError
  33. Is the sizeof(some pointer) always equal to four?
  34. What does it mean double free detected in tcache 2 while using mpz?
  35. -bash: fork: Cannot allocate memory
  36. How to Update/Drop a Hive Partition?
  37. connect to host localhost port 22: Connection refused
  38. What should be hadoop.tmp.dir ?
  39. How to clear memory to prevent “out of memory error” in excel vba?
  40. How do I determine the size of an object in Python?
  41. I get this error: “glibc detected”
  42. Is Python faster and lighter than C++?
  43. Hive’s unix_timestamp and from_unixtime functions
  44. What does ** mean in C?
  45. Does stack grow upward or downward?
  46. Difference between INNER JOIN and LEFT SEMI JOIN
  47. Fatal Error: Allowed Memory Size of 134217728 Bytes Exhausted (CodeIgniter + XML-RPC)
  48. Fatal error: Out of memory, but I do have plenty of memory (PHP)
  49. Destructor for a doubly-linked list that points to its value
  50. C: pointer to array of pointers to structures (allocation/deallocation issues)
  51. What is Sandboxed_process0 and why do i have so many instances of it listed in the memory usage for my app
  52. What is ‘PermSize’ in Java?
  53. What’s the difference between a word and byte?
  54. Hive Map-Join configuration mystery
  55. Where is the classpath set for hadoop
  56. malloc(sizeof(int)) vs malloc(sizeof(int *)) vs (int *)malloc(sizeof(int))
  57. Refactoring WordPress to improve memory performance [closed]
  58. How to configure WordPress to handle 75,000 pages?
  59. WP_Query leaking absurd amounts of memory
  60. How to use more than 256MB of memory in the admin?
  61. Is there a way to exclude the content from the post variable to save on RAM usage?
  62. Programmatically adding images to the media library with wp_generate_attachment_metadata randomly fails
  63. What’s the difference between WP_MEMORY_LIMIT and WP_MAX_MEMORY_LIMIT?
  64. What’s the ideal way to profile WordPress memory usage?
  65. PHP memory friendly alternative to get_posts()
  66. WordPress variables and memory
  67. Allowed memory size of 268435456 bytes exhausted (tried to allocate 7146491840906359738 bytes)
  68. Multiple WordPress sites eat up alot of RAM
  69. Decreasing the Memory Consumption of a WordPress Site?
  70. Memory needed to host 200 wordpress sites?
  71. PHP Memory Limit vs. WP Memory Limit
  72. $wpdb->get_results leaking memory
  73. WP_MEMORY_LIMIT didn’t work in wp-config
  74. Does a WP site consumes memory resources when there are NO visitors?
  75. Best way to diagnose which plug-ins/theme uses the most memory?
  76. How can WordPress be made more memory efficient?
  77. WordPress database becoming huge. How to analyze and optimize it? fear of running out of memory
  78. What is the most efficient way to search users by their display name?
  79. WordPress plugin installation error
  80. High CPU & Memory Spikes?
  81. WordPress media upload limit?
  82. wordpress upload http error?
  83. How to solve suspected memory issue in custom WordPress loop?
  84. Memory Leak in plugin action
  85. Excessive memory used by update_option()
  86. Allowed memory size exhausted. WordPress side solution
  87. Blank page for edit.php
  88. CPU usage: AJAX and Heartbeat API on admin pages
  89. Memory usage when querying users
  90. How can I free up the memory used by update_post_meta?
  91. Not enough memory
  92. Memory usage for scalable usermeta queries
  93. loop through all posts memory error
  94. Allowed Memory Size Error in WordPress
  95. How many caching plugins should be used?
  96. Mass importing json data into custom fields (postmeta)
  97. Memory errors with media upload, WordPress can’t use more than 96M (while there’s 512 available!)
  98. Can you limit the memory usage of a particular plugin?
  99. Are there individual memory allocations for different user roles in WordPress?
  100. Set WP_MAX_MEMORY_LIMIT higher than PHP.ini memory_limit

Leave a Comment

error code: 521