dev projects. Docker lets me focus on the actual project rather than wasting time on the
structure of the server (I’ll be discussing docker in greater detail in an upcoming article).
Docker is by no means a silver bullet and, like all services, there are some issues. The one
that I began to encounter was in the way the service carries out its logging. It’s a problem
that only affects ECS and resulted in many a sleepless night for me.
In order to preserve the logs and make them printable with docker commands, you need to
store the logs as a json file. The issue here is that the default log rotation prevents docker
from being able to read the log file and so after a relatively short period of time this log file
uses up all of your disk space.
If you’re wondering if there is a log rotate option that will work with docker, you would be
correct, however I haven't had the chance yet to test it and so for this article I’m going to
focus on the method I use to handle the problem.
When I first encountered the disk space issue I needed to hunt down the culprit that was
responsible for it. So I went the usual route and used the du command coupled with sort to
list the usages from lowest to highest.
du / | sort -nr
Ideally my culprit should have been the last entry in the list. Unfortunately, seeing as I was
using close to 100% of the available hard drive space, the sort wasn’t able to work.
I therefore needed a different method and, after some searching, I came across someone
with the same issue on Stack Exchange who had found a solution. Win!
The answer is to use a grep and filter out any files that are not bigger than 1G
du -h / | grep '[0-9\.]\+G'
This will print out the culprit which you can then delete and get back in business.
My next step is to figure out whether the log rotation function will work with docker so that
I won’t have to do this every so often.
References:
stackexchange - Tracking down where disk space has gone on Linux?man du - du(1)
