Docker: Container keeps on restarting again on again

Question

I today deployed an instance of MediaWiki using the appcontainers/mediawiki docker image, and I now have a new problem for which I cannot find any clue. After trying to attach to the mediawiki front container using:

docker attach mediawiki_web_1

which answers Terminated on my configuration for a reason I ignore, trying also:

docker exec -it mediawiki_web_1 bash

I do get something close to an error message:

Error response from daemon: Container 81c07e4a69519c785b12ce4512a8ec76a10231ecfb30522e714b0ae53a0c9c68 is restarting, wait until the container is running

And there is my new problem, because this container never stop restarting. I can see that using docker ps -a which always returns a STATUS of Restarting (127) x seconds ago.

The thing is, I am able to stop the container (I tested) but starting it again seems to bring it back into its restarting loop.

Any idea what could be the issue here ? The whole thing was properly working until I tried to attach to it...

I am sad :-(

Solution 1

The docker logs command will show you the output a container is generating when you don't run it interactively. This is likely to include the error message.

docker logs --tail 50 --follow --timestamps mediawiki_web_1

You can also run a fresh container in the foreground with docker run -ti <your_wiki_image> to see what that does. You may need to map some config from your docker-compose yml to the docker command.

I would guess that attaching to the media wiki process caused a crash which has corrupted something in your data.

Solution 2

When docker kill CONTAINER_ID does not work and docker stop -t 1 CONTAINER_ID also does not work, you can try to delete the container:

docker container rm CONTAINER_ID

I had a similar issue today where containers were in a continuous restart loop.

The issue in my case was related to me being a poor engineer.

Anyway, I fixed the issue by deleting the container, fixing my code, and then rebuilding and running the container.

Hope that this helps anyone stuck with this issue in future

Solution 3

tl;dr It is restarting with a status code of 127, meaning there is a missing file/library in your container. Starting a fresh container just might fix it.

Explanation:

As far as my understanding of Docker goes, this is what is happening:

1. Container tries to start up. In the process, it tries to access a file/library which does not exist.
2. It exits with a status code of 127, which is explained in this answer.
3. Normally, this is where the container should have completely exited, but it restarts.
4. It restarts because the restart policy must have been set to something other than no (the default), (using either the command line flag --restart or the docker-compose.yml key restart) while starting the container.

Solution: Something might have corrupted your container. Starting a fresh container should ideally do the job.

Solution 4

This could also be the case if you have created a systemd service that has:

[Service]
Restart=always
ExecStart=/usr/bin/docker container start -a my_container
ExecStop=/usr/bin/docker container stop -t 2 my_container

Solution 5

From personal experience it sounds like there is a problem within your docker container that is not allowing it to restart. So some process within the container is causing the restart to hang or some process is causing the container to crash on start.

When you start the container make sure you start it detached "-d" if you are going to attach to it. (ex. "docker run -d mediawiki_web_1")

View more solutions

Comments

[alberto56]

I had success by completely deleting my entire Docker cache, using forums.docker.com/t/how-to-delete-cache/5753/2 (I also added the -f tag to rmi). Then I rebuilt my containers and they worked.

[Katie Byers]

For me it wasn't enough to delete containers and images (as described in @alberto56's link), I also had to delete the associated volume. Once I did that, I was back in business.

[Balessan]

I assume running the container using docker-compose detached it anyway, no ? Or the -d argument is missing in my config file. will check that.

[Balessan]

Result of the command you provided which I guess is getting the last 50 logs related to the container is the following: 2016-05-26T16:38:27.362409489Z * Stopping web server apache2 * 2016-05-26T21:49:11.376549083Z Terminated 2016-05-26T21:49:11.688655642Z /bin/bash: /tmp/.runconfig.sh: No such file or directory, so you're right, there something corrupted in the data as the runconfig.sh seems to have disappeared. I will try to run the container once more from in the foreground as you advised. Just need to find how to specify the 25 proper arguments ^^

[Balessan]

Thanks, running a fresh container did the job. Docker was supposed to ease my deployment but for now it is a big failure :-) I probably need to learn and try more...

[Giannis Katsini]

I had put bad code in my application and in my docker compose file i added restart: always which left me in a loop of docker trying to start a broken app.. :(

[Blizzardengle]

I was pulling my hair out trying to get MySQL working. docker ps -a showed me that it was stuck in a boot loop and your command showed me why: files already in the mysql directory that it could not delete. You saved my from hours more of pulling my hair out. Thanks!

[Jimmy Adaro]

Why would you set "restart" multiple times in that service? Shouldn't it be done once?

[blissweb]

You sir, are the winner :-)

[jscul]

Ah, I see you're a poor engineer, I too am a poor engineer and had exit() in my code by accident.

[Mihir Bhatt]

every steps make sense in my case..

[SevakPrime]

You're absolutely right that it's better to fix the issue then the other answers given here. What would help is to show how one can find the logs since the container keeps restarting, so the logs are gone. Simply removing the restart parameter won't help either because the logs will still be gone when the container fails. If you update this with that info, I think you would get more upvotes.

[justnajm]

Thanks, actually there are some requirements by the container itself, and it helped out debugging what is missing, great help