CDH 4.1: Error running child : java.lang.OutOfMemoryError: Java heap space

hadoop mapreduce cloudera hadoop-yarn

13,944

Solution 1

Ok so seems like the problem is solved. Here is my mapred-site.xml

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>0.0.0.0:8021</value>
  </property>

  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

  <property>
    <description>To set the value of tmp directory for map and reduce tasks.</description>
    <name>mapreduce.task.tmp.dir</name>
    <value>/var/lib/hadoop-mapreduce/cache/${user.name}/tasks</value>
  </property>

  <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0.0:9290</value>
  </property>
  <property>
    <name>mapred.jobtracker.plugins</name>
    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    <description>Comma-separated list of jobtracker plug-ins to be activated.</description>
  </property>
  <property>
      <name>mapred.child.java.opts</name>
      <value>-Xmx512m</value>
  </property>
  <property>
      <name>io.sort.mb</name>
      <value>64</value>
  </property>
</configuration>

Here is my hadoop-env.sh:

export HADOOP_CLIENT_OPTS="-Xmx512m -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/cloudera/heapdump $HADOOP_CLIENT_OPTS"

I suppose the problem was in changes applying. Only restart helped me. I've tried to restart services - no effect.

Solution 2

One could also try setting the mapred.job.shuffle.input.buffer.percent to 20%. By default, this is set to 70%, which could be a lot if you are working on a very large set of data.

Try setting this property in mapred-site.xml:

<property>
   <name>mapred.job.shuffle.input.buffer.percent</name>
   <value>0.20</value>
</property>

Please refer to this document for some more performance tuning: http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf

Hope someone finds this useful :)

13,944

Author by

Capacytron

Updated on June 05, 2022

Comments

Capacytron almost 2 years

I've trying to overcome sudden problem. Befor that problem I've used old VM. I've downloaded the new one VM and still can't make my job run. I get Java heap space error. I've alredy read this one post: out of Memory Error in Hadoop

Here is my configs from /etc/hadoop/conf:

sudo vi hadoop-env.sh

# Extra Java runtime options.  Empty by default.
#export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS"
export HADOOP_CLIENT_OPTS="-Xmx256m $HADOOP_CLIENT_OPTS"

Here is my mapred-site.xml

<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx256m</value>
  </property>
  <property>
    <name>io.sort.mb</name>
    <value>128</value>
  </property>

Nothing helps :(

Here is the console output:

Deleted /var/log/hadoop-yarn/apps/cloudera/logs
12/12/02 16:31:45 INFO input.FileInputFormat: Total input paths to process : 1
12/12/02 16:31:45 INFO mapreduce.JobSubmitter: number of splits:13
12/12/02 16:31:45 WARN conf.Configuration: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
12/12/02 16:31:45 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
12/12/02 16:31:45 WARN conf.Configuration: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
12/12/02 16:31:45 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
12/12/02 16:31:45 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
12/12/02 16:31:45 WARN conf.Configuration: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
12/12/02 16:31:45 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
12/12/02 16:31:46 INFO mapred.ResourceMgrDelegate: Submitted application application_1354455034384_0007 to ResourceManager at /0.0.0.0:8032
12/12/02 16:31:46 INFO mapreduce.Job: The url to track the job: http://localhost.localdomain:8088/proxy/application_1354455034384_0007/
12/12/02 16:31:46 INFO mapreduce.Job: Running job: job_1354455034384_0007
12/12/02 16:31:51 INFO mapreduce.Job: Job job_1354455034384_0007 running in uber mode : false
12/12/02 16:31:51 INFO mapreduce.Job:  map 0% reduce 0%
12/12/02 16:32:02 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000005_0, Status : FAILED

Killed by external signal

12/12/02 16:32:15 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000006_0, Status : FAILED
Error: Java heap space
12/12/02 16:32:19 INFO mapreduce.Job:  map 1% reduce 0%
12/12/02 16:32:29 INFO mapreduce.Job:  map 2% reduce 0%
12/12/02 16:32:29 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000005_1, Status : FAILED
Error: Java heap space
12/12/02 16:32:36 INFO mapreduce.Job:  map 3% reduce 0%
12/12/02 16:32:40 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000006_1, Status : FAILED
Error: Java heap space
12/12/02 16:32:43 INFO mapreduce.Job:  map 4% reduce 0%
12/12/02 16:32:51 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000005_2, Status : FAILED
Error: Java heap space
12/12/02 16:32:53 INFO mapreduce.Job:  map 5% reduce 0%
12/12/02 16:33:00 INFO mapreduce.Job:  map 6% reduce 0%
12/12/02 16:33:03 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000006_2, Status : FAILED
Error: Java heap space
12/12/02 16:33:07 INFO mapreduce.Job:  map 7% reduce 0%
12/12/02 16:33:15 INFO mapreduce.Job:  map 8% reduce 0%
12/12/02 16:33:15 INFO mapreduce.Job:  map 15% reduce 0%
12/12/02 16:33:15 INFO mapreduce.Job: Job job_1354455034384_0007 failed with state FAILED due to: 
12/12/02 16:33:15 INFO mapreduce.Job: Counters: 31
    File System Counters
        FILE: Number of bytes read=600
        FILE: Number of bytes written=349925
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=105310577
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=15
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
    Job Counters 
        Failed map tasks=7
        Launched map tasks=12
        Other local map tasks=5
        Data-local map tasks=7
        Total time spent by all maps in occupied slots (ms)=597080
        Total time spent by all reduces in occupied slots (ms)=0
    Map-Reduce Framework
        Map input records=252675
        Map output records=15
        Map output bytes=992
        Map output materialized bytes=0
        Input split bytes=590
        Combine input records=0
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=272051
        CPU time spent (ms)=112790
        Physical memory (bytes) snapshot=1094082560
        Virtual memory (bytes) snapshot=3678527488
        Total committed heap usage (bytes)=780533760
    File Input Format Counters 
        Bytes Read=105938372

Here is my log:

2012-12-02 16:36:13,072 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2012-12-02 16:36:14,405 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2012-12-02 16:36:15,552 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1354455034384_0008
2012-12-02 16:36:15,916 WARN [main] org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2012-12-02 16:36:15,919 WARN [main] org.apache.hadoop.conf.Configuration: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
2012-12-02 16:36:15,920 WARN [main] org.apache.hadoop.conf.Configuration: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
2012-12-02 16:36:15,920 WARN [main] org.apache.hadoop.conf.Configuration: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2012-12-02 16:36:15,921 WARN [main] org.apache.hadoop.conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
2012-12-02 16:36:15,922 WARN [main] org.apache.hadoop.conf.Configuration: job.local.dir is deprecated. Instead, use mapreduce.job.local.dir
2012-12-02 16:36:15,922 WARN [main] org.apache.hadoop.conf.Configuration: mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files
2012-12-02 16:36:15,922 WARN [main] org.apache.hadoop.conf.Configuration: mapred.job.id is deprecated. Instead, use mapreduce.job.id
2012-12-02 16:36:16,575 WARN [main] org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
2012-12-02 16:36:18,332 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@3582c132
2012-12-02 16:36:21,168 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:912)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:638)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:709)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)

2012-12-02 16:36:21,448 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2012-12-02 16:36:21,448 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2012-12-02 16:36:21,448 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
(&container_1354455034384_0008_01_000005�}stderr156WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
stdout0syslog35152012-12-02 16:35:31,838 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2012-12-02 16:35:32,283 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-12-02 16:35:32,283 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2012-12-02 16:35:33,314 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2012-12-02 16:35:34,339 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1354455034384_0008
2

What can I try next? Please help. Thank you.