How to write logs to a file using Log4j and Storm Framework?

Question

I am having bit of an issue in logging to a file using log4j in storm .

Before submitting my topology , i.e in my main method I wrote some log statements and configured the logger using :

PropertyConfigurator.configure(myLog4jProperties)

• Now when I run my topology using my executable jar in eclipse - its working fine and log files are being created as supposed.
  OR
  When i run my executable jar using "java -jar MyJarFile someOtherOptions", i can see log4j being configured and the files are formed correctly and logging is done on both files and console (as defined in my log4j.properties)

• BUT when i run the same jar using "storm jar MyJarFile MyMainClass someOtherOptions" it is not being able to create and log into any of the files except on console.

I am talking about the logs I am printing BEFORE submitting my topology.

Is there any way to log my statements in a file while using storm ? I am not bound to use org.apache.log4j.

Solution 1

The storm framework uses its own logging. Your logs most likely will end up in the logs dir where storm is installed({Storm DIR}/logs). You can find the storm log properties in {Storm DIR}/logback/cluster.xml. It uses logback not log4j

Solution 2

I would recommend using SLF4J for your logging within storm. You probably could get LOG4J working but you will have additional setup to do on each node in the cluster. Since storm does this for you already, I don't really see the point.

In your project, include these maven dependencies (slf4j-simple for your unit tests!):

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>1.7.5</version>
</dependency>
<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-simple</artifactId>
  <version>1.7.5</version>
  <scope>test</scope>
</dependency>

Then, inside your topologies/spouts/bolts you just get the Logger:

private static final Logger LOG = LoggerFactory.getLogger(MySpout.class);

In my environment, anything INFO and above is logged to file and, more importantly, visible in the Storm UI. You just need to click on your spout/bolt and then click on the port number to go to the log viewer page.

If you want to get to the actual files then you can gather them off of each node (mine are in /var/log/storm/).