Clickhouse fails to start and always exits after starting for a while. Not available.

[email protected] : systemctl status clickhouse-server
clickhouse - server . service - ClickHouse Server ( analytic DBMS for big data ) Main PID : 4895 ( code = exited , status = 0 / SUCCESS )
[email protected] : journalctl - u clickhouse - server
  
clickhouse - server [ 986 ]: Cannot add message to the log : Code : 243 , e . displayText () = DB :: Exception : Cannot reserve 1.00 MiB , not enough space , Stack t

    You can also go to clickhouse's default log directory /var/log/clickhouse-server to check. After sorting out, there is a line in the top log: not enough space. Then check the disk and find that the system disk is indeed 100% used. Through investigation, it is found that the containers under /var/lib/docker/containers occupy a large space, and the log file is nearly 100G in depth. Of course, the log can be cleared to reduce the occupancy. But the same problem still occurs next time.

The root of solving the excessively large docker container log is to control the size of the container log. It is necessary to control the log size when creating the container. You can use the following parameters to achieve.

[email protected]: docker run -it --log-opt max-size=100m --log-opt max-file=10 clickhouse-server

The best way is to modify the docker configuration file /etc/docker/daemon.json, and add the configuration as follows to restart docker and recreate the container to take effect:

[email protected]:# cat /etc/docker/daemon.json
{
     "log-driver": "json-file",
     "log-opts": {
         "max-size" : "100m",
         "max-file": "10"
     }
}
[email protected]: # systemctl daemon-reload
[email protected]: # systemctl restart docker

    In addition, we need to troubleshoot the problem of the container. If the container log is too large, it may be caused by a problem with the container startup that causes the container to report errors and write logs. Log files can be found for troubleshooting.