Strange hang-ups with Docker

Today I had to change some buffer setting for MySQL in my.cnf.

So I set the new settings and tried /etc/init.d/mysql reload.

This passed successfully but when I went to the running mysql to check the settings they weren’t changed.

So I decided to make /etc/init.d/mysql restart.

Big mistake because the mysqld process was keeping the container from closing. I suspect it stopped mysql and then the container stopped too, there was only one additional sshd process running which I used to connect to the container.

However the DB is in good condition, but what happened after the mysql container closed is bothering me. That’s why they say don’t use it in production! 😉

I’ve executed docker start mysql_container

It started, atop showed me a huge hdd usage for a second and the port was open but I couldn’t connect with telnet or the mysql client. I’ve made one docker commit –run=” mysql_container iliyan/mysql and kill/rm of the current one. Checked the mysql port – it was closed, nothing hanged there. Then ran the iliyan/mysql with bash, started the mysql server from there it started successfully, connecting with the mysql client locally to the server succeeded and then I realized it’s something controlled by Docker that hanged.

So I decided to restart the docker process which restart all of the containers (service docker restart). It restarted, started 3 containers from 5 and I ha to just docker start cont4 cont5 to finish the process.

Now everything is working well again but this one have to be monitored. Who know if this is not going to happen again or every time a long running container is restarted.

Update: Some time passed and the things were mostly good. However from yesterday I wonder why the hell the page visits lowered to one of my site and decided to check the firewall:

iptables -L

What I saw from the first look is that I have too much repeating IPs of the containers. I also remember that the last time I restarted the mysql percona container (I keep a big DB inside, no volumes, so this will change soon) the percona and git/gitweb containers were started but not reachable, I couldn’t start them again, restart works but no ports were set, no IP seen in `docker ps`, etc. Then I’ve made `service docker restart` again and the contiatiners were working but I better check the firewall and the system as a whole when changing anything in the containers, also `docker ps -a`, `docker top cont_name` and `docker logs cont_name` and rebooting the whole host server to fix it if not possible either way. I still don’t know a lot of things about system administration, still I think this problem comes from 1-2 containers that have an exotic way of installation/configuration inside. I will definitely check them one by one and probably when docker 1.0 is here I will have fixed my problem as well! 🙂

Update2:

After I changed the process holding the container, using mostly the `tail -f filename.log` command, the logs produced for `docker log containername` are coming more often and after a few days I still have good results for `docker top containername` on the troubled containers which are more than fine now. Probably if there is not a log for a long time the container will show you nothing when executing `docker top` on it. We’ll see.

Let’s be patient and continue testing this great tool!

Put your GitLab on HTTPS

For this article I will use the following configuration:GitLab Docker image by sameersbn ...

Learn more

About Iliyan Trifonov

Web Developer, Blogger
Tagged , , , , , , , . Bookmark the permalink.