Where does Hadoop store the logs of YARN applications?

Question

I run the basic example of Hortonworks' yarn application example. The application fails and I want to read the logs to figure out why. But I can't find any files at the expected location (/HADOOP_INSTALL_FOLDER/logs) where the logs of my mapreduce jobs are stored.

Does anybody know where yarn stores the non-mapreduce log files? Do I have to configure a special directory in the xml files?

Solution 1

The container logs should be under yarn.nodemanager.log-dirs:

Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.

Solution 2

Log-aggregation has been implemented in YARN, because of which the log file locations will vary when compared with Hadoop 1.

You can get the logs of your application in two ways, WebUi and Command line access.

Please go through the below document which gives you a very clear information on this log-aggregation implementation on YARN.

http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/

Thanks to the blog author, Vinod Kumar Vavilapalli.

Comments

[padmalcom]

This is exactly what I was looking for. Setting YARN_LOG_DIR in yarn-env.sh is not necessary when you set yarn.nodemanager.log-dirs to an absolute path. Alternatively, you can set only YARN_LOG_DIR and leave yarn.nodemanager.log-dirs at its default value ${YARN_LOG_DIR}/logs to set the YARN log directory.

[Prashanth]

Hi, how do I delete these yarn logs? Does removing the yarn log file does it? I'm totally new to yarn..

[Remus Rusanu]

@Prashanth ask questions as separate questions, not as comments.