How do I increase Tez's container physical memory?

Question

I've been running some hive scripts on an aws emr 4.8 cluster with hive 1.0 and tez 0.8.

My configurations look like this:

SET hive.exec.compress.output=true;
SET mapred.output.compression.type=BLOCK;
SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;
set hive.execution.engine=tez;
set hive.merge.mapfiles=false;
SET hive.default.fileformat=Orc;
set tez.task.resource.memory.mb=5000;
SET hive.tez.container.size=6656;
SET hive.tez.java.opts=-Xmx5120m;
set hive.optimize.ppd=true;

And my global configs are:

hadoop-env.export   HADOOP_HEAPSIZE 4750
hadoop-env.export   HADOOP_DATANODE_HEAPSIZE    4750
hive-env.export HADOOP_HEAPSIZE 4750

While running my script, I get the following error:

Container [pid=19027,containerID=container_1477393351192_0007_02_000001] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.9 GB of 5 GB virtual memory used. Killing container.

On googling this error, I read that set tez.task.resource.memory.mb will change the physical memory limit, but clearly I was mistaken. What am I missing?

Solution 1

I have had this problem a lot. The changing

Set hive.tez.container.size=6656;
Set hive.tez.java.opts=-Xmx4g;

does not fix the problem for me but this does:

set tez.am.resource.memory.mb=4096;

Solution 2

Set the Tez container size to be a larger multiple of the YARN container size (4GB):

SET hive.tez.container.size=4096MB

"hive.tez.container.size" and "hive.tez.java.opts" are the parameters that alter Tez memory settings in Hive. If "hive.tez.container.size" is set to "-1" (default value), it picks the value of "mapreduce.map.memory.mb". If "hive.tez.java.opts" is not specified, it relies on the "mapreduce.map.java.opts" setting. Thus, if Tez specific memory settings are left as default values, memory sizes are picked from mapreduce mapper memory settings "mapreduce.map.memory.mb".

https://documentation.altiscale.com/memory-settings-for-tez

For more info Tez configuration and Tez memory tuning

Note: Set in MB with Ambari

Solution 3

Incase anyone else stumbles upon this thread trying to solve this above, here is a link to a real solution that worked for me where all the other solutions did not.

http://moi.vonos.net/bigdata/hive-cli-memory/

TL;DR add these to your hive call --hiveconf tez.am.resource.memory.mb=<size as int> --hiveconf tez.am.launch.cmd-opts=""

Comments

[jackStinger]

SO how is it different from SET hive.tez.container.size=6656; SET hive.tez.java.opts=-Xmx5120m;

[jackStinger]

Is there a reason SET hive.tez.container.size=6656; won't work?

[jackStinger]

Also, Query returned non-zero code: 1, cause: 'SET hive.tez.container.size=6656MB' FAILED because hive.tez.container.size expects INT type value.

[jackStinger]

So I had set the container size, as well as the tez.java.opts in my configurations already. Despite that, my physical memory is 1G. The altiscale page is exactly where I found these, but didn't work for me. :/

[user3062149]

There are no semicolons at the end of the statements. I had a similar failure "... expects INT type value" with container.size. I removed the semicolon at the end of the statement and the failure went away. But I have no idea why that fix works.

[Reihan_amn]

does this commands work on hadoop conf? or just for hive? should this works for example (set mapreduce.reduce.memory.mb = 4096)?

[Reihan_amn]

does this commands work on hadoop conf? or just for hive? should this works for example (set mapreduce.reduce.memory.mb = 4096)?

[StanleyZ]

tez.am.resource.memory.mb=4096; is the solution for me. in my case it's the app master (am) has too little memory so the job failed when scheduling.