Sudden Xibo SQL Error

We suddenly started getting this error from our CMS server. Semi recently updated the server and CMS but we thought it was fine in the immediate aftermath maybe we missed it though. We are using Docker. Reboot didn’t fix it. Thoughts? Thank you for your time.

Fatal Error - sorry this shouldn’t happen. SQLSTATE[HY000] [2005] Unknown MySQL server host ‘mysql’ (110)

So it’s saying it can’t connect to the MySQL database.

First thing to check is if the database container is running, so run

docker ps -a

What does that output?

Output is as follows, and yes that status time for the mysql definitly seems wrong next to the others, not sure what it means.

CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                    PORTS                               NAMES
bd52dbb498fc        xibosignage/xibo-cms:release_1.8.2   "/entrypoint.sh"         5 days ago          Up 16 hours               0.0.0.0:80->80/tcp                  xibodocker_cms-web_1
12ee1795ece0        mysql:5.6                            "docker-entrypoint..."   5 days ago          Up 8 seconds              3306/tcp                            xibodocker_cms-db_1
fd36da0d0ad3        xibosignage/xibo-xmr:release_1.8.2   "/entrypoint.sh"         5 days ago          Up 16 hours               0.0.0.0:9505->9505/tcp, 50001/tcp   xibodocker_cms-xmr_1
a5f8357a4263        hello-world                          "/hello"                 2 months ago        Exited (0) 2 months ago                                       agitated_chandrasekhar

So the MySQL container is showing as up for 8 seconds, which probably means it’s restarting over and over.

What do the logs for it show?

docker-compose logs

Oh my, this over and over again, a lot.

cms-db_1   | 2017-07-18 13:25:37 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2017-07-18 13:25:37 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...
cms-db_1   | 2017-07-18 13:26:47 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2017-07-18 13:26:47 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...
cms-db_1   | 2017-07-18 13:27:58 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2017-07-18 13:27:58 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...

How much RAM has your host machine got available?

You could try recreating the containers - so run

docker-compose down
docker-compose up -d

If you used a custom compose file, then be sure to add in that as part of the command - eg

docker-compose -f cms_custom-ports.yml down
docker-compose -f cms_custom-ports.yml up -d

Does that give any errors?

Effectively as much as it wants. It is a Hyper-V instance with Dynamic RAM turned on. (It is Ubuntu server OS on the VM.) But that said, it starts at 512MBs and scales up from there. Currently, it reads as having claimed 2710MBs for itself.

OK that should be fine. Please see my edits to my previous post.

I am not using a custom file, trying to keep it all very “stock” at the moment.

OK sure. So what happens if you recreate the containers?

docker-compose down
docker-compose up -d
docker-compose logs

Sorry, sorry, bad read on my part.

Attaching to xibodocker_cms-web_1, xibodocker_cms-db_1, xibodocker_cms-xmr_1
cms-web_1  | Waiting for MySQL to start - max 300 seconds
cms-web_1  | MySQL started
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | New install
cms-web_1  | Provisioning Database
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | Configuring Database Settings
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | Setting up Maintenance
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | ERROR 2003 (HY000): Can't connect to MySQL server on 'mysql' (111)
cms-web_1  | Updating settings.php
cms-web_1  | Configuring Backups
cms-web_1  | Configuring Maintenance
cms-web_1  | Running maintenance
cms-web_1  | Starting cron
cms-web_1  | Starting webserver
cms-web_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.4. Set the 'ServerName' directive globally to suppress this message
cms-db_1   | 2017-07-18 13:43:07 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2017-07-18 13:43:07 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...
cms-db_1   | 2017-07-18 13:43:15 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2017-07-18 13:43:15 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...
cms-db_1   | 2017-07-18 13:43:22 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

Odd. Your config.env file is still there I take it (and has the original MySQL password you chose set)? And shared/db still contains all your database files?

What are the permissions on the shared/db directory and files below it?

From my install, the permissions from the host are as follows:

alex@alex-desktop:~/shared/db$ ls -al
total 176152
drwxr-xr-x 5  999 docker     4096 Jul 18 07:47 .
drwxr-xr-x 5 root root       4096 Feb 27 09:11 ..
-rw-rw---- 1  999 docker       56 Feb 28 09:17 auto.cnf
drwx------ 2  999 docker     4096 Jun 26 15:06 cms
-rw-rw---- 1  999 docker 79691776 Jul 18 14:50 ibdata1
-rw-rw---- 1  999 docker 50331648 Jul 18 14:50 ib_logfile0
-rw-rw---- 1  999 docker 50331648 Feb 28 09:17 ib_logfile1
drwx------ 2  999 docker     4096 Jun 26 15:02 mysql
drwx------ 2  999 docker     4096 Jun 26 15:02 performance_schema

I’d expect yours to look the same, including the owner to be 999 (unless your host machine has a user with that ID). Similarly, I’d expect the same files to be present etc. What do you have?

config.env is still in place, the settings appear to be the same as they have been, our password is a long random string so its hard to be 100% but other custom mail settings are all there. And our permissions seem the same.

localadmin@xibo:/srv/xibo-docker/shared/db$ ls -al
total 634908
drwxr-xr-x 5        999 docker          4096 Jul 18 08:56 .
drwxrwxr-x 5 localadmin localadmin      4096 May  8 09:20 ..
-rw-rw---- 1        999 docker            56 May  8 09:20 auto.cnf
drwx------ 2        999 docker          4096 Jul 12 11:29 cms
-rw-rw---- 1        999 docker     549453824 Jul 18 08:53 ibdata1
-rw-rw---- 1        999 docker      50331648 Jul 18 08:56 ib_logfile0
-rw-rw---- 1        999 docker      50331648 Jul 18 08:56 ib_logfile1
drwx------ 2        999 docker          4096 May  8 09:20 mysql
drwx------ 2        999 docker          4096 May  8 09:20 performance_schema

That all looks reasonable then. You already rebooted so I don’t see any benefit in restarting the Docker service.

Is this running on something I can get remote access to? If so, PM me the details and I’ll have a look. I can send you a public SSH key if that’s easier than giving me username/password access over SSH.

Alternatively, tar up the shared/db folder and send it over to me, and I’ll see if I can get it running here. You’ll need to upload the tarball somewhere and PM me the link. I’m heading out shortly but will be back later tonight or tomorrow.

Can you just check your /var/log/syslog file for any lines that contain “Out of memory”.

eg

cat /var/log/syslog | grep "Out of memory"

Were there any Out of Memory lines in your syslog?

For anyone who comes to this, did some PMing with Alex and it currently appears our database corrupted and we are looking at restoring from the last good backup.

I know this is an old post but I ran into this problem when I ran xibo in my docker swarm I added below to my config.env / cms-web and updated my stack. It worked.
(Fill in your own info.)
MYSQL_HOST=
MYSQL_PORT=3306
MYSQL_DATABASE=
MYSQL_PASSWORD=
MYSQL_USER=

@alex
I’m having same error. Tried everything but still getting error saying
Fatal Error - sorry this shouldn’t happen. SQLSTATE[HY000] [2002] Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)

MySQL is trying to connect via socket rather than IP. I guess you’ve got “localhost” for MYSQL_HOST?

That won’t work, because localhost for the CMS container is itself, and not your host machine. You need to put in the local network name or IP of the machine that is running your MySQL server.