Docker container with broken Mysql database. Lesson learned. Backup often. Backup your DB or at least the container.

I’ve been updating lxc-docker every time a new version is suggested by my Linux OS. While the apt-get install lxc-docker was working it stops and starts again with the new Docker version. Usually nothing happens with the containers being forced stop and started that way but today I’ve had a broken Mysql DB for this WordPress installation and couldn’t restore it from its data directory no matter what I’ve tried. Because I didn’t want to loose more time I decided I can restore previous commits to the same image that container is running from.

I’ve found the docker history imagename command and tried it. I’ve added a couple of commits to this image and I’ve had a couple of lines with the history commands. I’ve picked the commit before the broken state and ran the wordpress container the same way I am running it with the :latest tag but this time used the commit ID. It ran well and I’ve made a commit from it to its image. Stopped and removed the wordpress container and ran it with the usual command using the latest imagename command this time and now it’s running well.

Now I know that when I am going to stop the images and update the Docker executable I must backup the databases, files and commit the latest state of the containers (not much by the rules because the container should be able to be rebuilt at any time keeping the data outside the container).

Strange hang-ups with Docker

Today I had to change some buffer setting for MySQL in my.cnf.

So I set the new settings and tried /etc/init.d/mysql reload.

This passed successfully but when I went to the running mysql to check the settings they weren’t changed.

So I decided to make /etc/init.d/mysql restart.

Big mistake because the mysqld process was keeping the container from closing. I suspect it stopped mysql and then the container stopped too, there was only one additional sshd process running which I used to connect to the container.

However the DB is in good condition, but what happened after the mysql container closed is bothering me. That’s why they say don’t use it in production! 😉

I’ve executed docker start mysql_container

It started, atop showed me a huge hdd usage for a second and the port was open but I couldn’t connect with telnet or the mysql client. I’ve made one docker commit –run=” mysql_container iliyan/mysql and kill/rm of the current one. Checked the mysql port – it was closed, nothing hanged there. Then ran the iliyan/mysql with bash, started the mysql server from there it started successfully, connecting with the mysql client locally to the server succeeded and then I realized it’s something controlled by Docker that hanged.

So I decided to restart the docker process which restart all of the containers (service docker restart). It restarted, started 3 containers from 5 and I ha to just docker start cont4 cont5 to finish the process.

Now everything is working well again but this one have to be monitored. Who know if this is not going to happen again or every time a long running container is restarted.

Update: Some time passed and the things were mostly good. However from yesterday I wonder why the hell the page visits lowered to one of my site and decided to check the firewall:

iptables -L

What I saw from the first look is that I have too much repeating IPs of the containers. I also remember that the last time I restarted the mysql percona container (I keep a big DB inside, no volumes, so this will change soon) the percona and git/gitweb containers were started but not reachable, I couldn’t start them again, restart works but no ports were set, no IP seen in `docker ps`, etc. Then I’ve made `service docker restart` again and the contiatiners were working but I better check the firewall and the system as a whole when changing anything in the containers, also `docker ps -a`, `docker top cont_name` and `docker logs cont_name` and rebooting the whole host server to fix it if not possible either way. I still don’t know a lot of things about system administration, still I think this problem comes from 1-2 containers that have an exotic way of installation/configuration inside. I will definitely check them one by one and probably when docker 1.0 is here I will have fixed my problem as well! 🙂

Update2:

After I changed the process holding the container, using mostly the `tail -f filename.log` command, the logs produced for `docker log containername` are coming more often and after a few days I still have good results for `docker top containername` on the troubled containers which are more than fine now. Probably if there is not a log for a long time the container will show you nothing when executing `docker top` on it. We’ll see.

Let’s be patient and continue testing this great tool!

How to share a common IP between Docker containers

Sometimes you may build your images manually and don’t need something special but something to be used asap.

Such case happened when I wanted to tell php where is the mysql server. Usually after creating and removing containers you may see the mysql container’s ip changed and you have to change the configuration in your site again.

I’ve decided to use one approach that helps me set and forget about the sites’s configuration files and this is using a common IP inside my Docker host that is not 127.0.0.1 or 0.0.0.0.

I’ve made one ifconfig and I’ve seen these lines:

docker0 inet addr:172.17.42.1

lxcbr0 inet addr:10.0.3.1

I just used docker run …… -p 10.0.3.1:3306:3306 …/mysql

Then in the sites’s configs the mysql host was given as 10.0.3.1 instead of 127.0.0.1 or 172.17.0.X

That was since Docker version 0.9.0 went out. I now see only the docker0 ip which still solves the problem the same way with different common IP. I am using libcontainer’s API, no -e lxc, only custom docker data and temp directories set in the defaults file.