ActiveMQ won't restart because KahaDB is locked

Question

Locally to start/stop ActiveMQ (5.6) on my dev machine I just run ./activemq start and ./activemq stop respectively.

On our QA machine we have it installed as a service and run service activemq start and service activemq stop respectively.

I just went to bounce the QA machine and issued service activemq stop, then service activemq start to restart it.

I see a process ID being created, and if I run ps -aef | grep activemq I see the living, breathing process of an ActiveMQ server.

But if I go to http://<qa-server>:8161/admin I get the typical error that you see when a server is down:

Firefox can't establish a connection to the server at :8161.

Edit: I have now tried both the ./active start and service activemq start methods, and both produce the same issue: I see a process being created, but nothing in the web admin tool.

I checked ActiveMQ's home directory and don't see any type of logs/ directory, so I'm not even sure where to begin debugging the issue.

Either AMQ is not restarting, or its web admin app isn't restarting or functioning properly; either way I have no idea where to start. Thanks in advance!

Edit:

I see the following error in data/activemq.log:

2012-10-07 11:37:14,501 | INFO | Database /qa-server/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/qa-server/kahadb/lock' could not be locked. | org.apache.activemq.store.kahadb.MessageDatabase | main 2012-10-07 11:37:24,504 | INFO | Database /qa-server/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/qa-server/kahadb/lock' could not be locked. | org.apache.activemq.store.kahadb.MessageDatabase | main

Solution 1

Turns out there were multiple AMQ servers in our QA environment. When I shutdown the first server, an exception was thrown for some reason and so it didn't release the lock. Possession of that lock then went to the other AMQ instance (the first server was the master, the 2nd server was the slave).

When I tried restarting the first server (the master), it wouldn't restart because the 2nd server had possession of the lock. I shutdown the 2nd server and the lock was released, allowing me to restart them both.

Solution 2

Check the data/ directory for logs.

And is this a typo? :8161.admin
Try :8161/admin

Solution 3

1. Go to your apache installation home folder in Win7 i.e. apache-activemq-X.XX.X
2. Right click and select properties
3. Change the access permission of this folder for your logged in user - "normally your logged in user should already have ADMIN access, but if you have this apache MQ unzipped on your machine, then admin permission would be missing"

Now, you should have activemq command running smoothly