History, Logging, and Debugging (SSH, The Secure Shell: The Definitive Guide)

History, Logging, and Debugging (SSH, The Secure Shell: The Definitive Guide).

Found this article while searching for a way to keep the sshd process in the foreground but it’s closed after the client is disconnected. However the article is about logging and is very informative.

Strange hang-ups with Docker

Today I had to change some buffer setting for MySQL in my.cnf.

So I set the new settings and tried /etc/init.d/mysql reload.

This passed successfully but when I went to the running mysql to check the settings they weren’t changed.

So I decided to make /etc/init.d/mysql restart.

Big mistake because the mysqld process was keeping the container from closing. I suspect it stopped mysql and then the container stopped too, there was only one additional sshd process running which I used to connect to the container.

However the DB is in good condition, but what happened after the mysql container closed is bothering me. That’s why they say don’t use it in production! 😉

I’ve executed docker start mysql_container

It started, atop showed me a huge hdd usage for a second and the port was open but I couldn’t connect with telnet or the mysql client. I’ve made one docker commit –run=” mysql_container iliyan/mysql and kill/rm of the current one. Checked the mysql port – it was closed, nothing hanged there. Then ran the iliyan/mysql with bash, started the mysql server from there it started successfully, connecting with the mysql client locally to the server succeeded and then I realized it’s something controlled by Docker that hanged.

So I decided to restart the docker process which restart all of the containers (service docker restart). It restarted, started 3 containers from 5 and I ha to just docker start cont4 cont5 to finish the process.

Now everything is working well again but this one have to be monitored. Who know if this is not going to happen again or every time a long running container is restarted.

Update: Some time passed and the things were mostly good. However from yesterday I wonder why the hell the page visits lowered to one of my site and decided to check the firewall:

iptables -L

What I saw from the first look is that I have too much repeating IPs of the containers. I also remember that the last time I restarted the mysql percona container (I keep a big DB inside, no volumes, so this will change soon) the percona and git/gitweb containers were started but not reachable, I couldn’t start them again, restart works but no ports were set, no IP seen in `docker ps`, etc. Then I’ve made `service docker restart` again and the contiatiners were working but I better check the firewall and the system as a whole when changing anything in the containers, also `docker ps -a`, `docker top cont_name` and `docker logs cont_name` and rebooting the whole host server to fix it if not possible either way. I still don’t know a lot of things about system administration, still I think this problem comes from 1-2 containers that have an exotic way of installation/configuration inside. I will definitely check them one by one and probably when docker 1.0 is here I will have fixed my problem as well! 🙂

Update2:

After I changed the process holding the container, using mostly the `tail -f filename.log` command, the logs produced for `docker log containername` are coming more often and after a few days I still have good results for `docker top containername` on the troubled containers which are more than fine now. Probably if there is not a log for a long time the container will show you nothing when executing `docker top` on it. We’ll see.

Let’s be patient and continue testing this great tool!

Setup your time on the host and inside the Docker containers

If you want to set the server’s timezone and keep it syncronized and you are using Ubuntu like me you can do the following:

On the host you can change the timezone and configure a cron to syncronize it:

dpkg-reconfigure tzdata

Pick your continent and capitol and save;

Then setup a cron to update the time daily:

echo "ntpdate ntp.ubuntu.com" > /etc/cron.daily/ntpdate && chmod 755 /etc/cron.daily/ntpdate

That’s it!

Now if you want to change the timezone inside a Docker container:

I usually have sshd running inside the containers which allows me to ssh inside them and make changes as well as reload the server process running inside to activate the changes:

docker inspect mysql_container

Now you see the ip of the container, for example: 172.17.0.2 and you use it here:

ssh 172.17.0.2

Update: now you can use docker exec to enter a container environment. Forget about running sshd server in docker containers:

docker exec -ti mycontainername bash
export TERM=xterm
#do your stuff here

Inside the container:

dpkg-reconfigure tzdata

You are not allowed to use ntpdate inside the container but it probably gets the time from the host(I still have some things to learn) and you only need the timezone set.

Another(prefered) way to make changes inside the container and keep them is to run another container from the image that the one you are going to change is running from.

docker run -ti --name mysql_container_tmp iliyan/mysql bash

Then make your changes, exit and commit the new container over the same image. Stop and remove the original container as well as the _tmp one and run a new container from the updated image. Now your changes are there!

Check this Dockerfile that automates the process (the part on the bottom that works with timezones) and also this bash script.

How to share a common IP between Docker containers

Sometimes you may build your images manually and don’t need something special but something to be used asap.

Such case happened when I wanted to tell php where is the mysql server. Usually after creating and removing containers you may see the mysql container’s ip changed and you have to change the configuration in your site again.

I’ve decided to use one approach that helps me set and forget about the sites’s configuration files and this is using a common IP inside my Docker host that is not 127.0.0.1 or 0.0.0.0.

I’ve made one ifconfig and I’ve seen these lines:

docker0 inet addr:172.17.42.1

lxcbr0 inet addr:10.0.3.1

I just used docker run …… -p 10.0.3.1:3306:3306 …/mysql

Then in the sites’s configs the mysql host was given as 10.0.3.1 instead of 127.0.0.1 or 172.17.0.X

That was since Docker version 0.9.0 went out. I now see only the docker0 ip which still solves the problem the same way with different common IP. I am using libcontainer’s API, no -e lxc, only custom docker data and temp directories set in the defaults file.

Docker 0.9: introducing execution drivers and libcontainer | Docker Blog

Docker 0.9: introducing execution drivers and libcontainer | Docker Blog.

It’s here and the quality is visible! 🙂

One thing to know coming from Docker 0.8.1 is if you want the old functionality, try this:

docker -d -e lxc

Using the defaults (/etc/default/docker):

DOCKER_OPTS=”-dns 8.8.8.8 -dns 8.8.4.4 -g /yourcustomdir -e lxc

See my link for the Docker’s changelog here.