Docker installation doesn't start anymore

Get Help
1

Hi,

We’ve got xibo 1.8.1 on docker. We’ve used it for a couple of months. But just before our holidays we had an power issue and our xibo setup doesn’t start up anymore.

extract from docker-compose logs:

@display:/opt/xibo# docker-compose logs | more
Attaching to xibo_cms-web_1, xibo_cms-xmr_1, xibo_cms-db_1
cms-db_1   | 2017-08-29 11:37:50 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documen
tation for more details).
cms-db_1   | 2017-08-29 11:37:50 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...
cms-web_1  | Waiting for MySQL to start - max 300 seconds
cms-web_1  | MySQL started
cms-web_1  | DBVersion
cms-web_1  | 132
cms-web_1  | Updating settings.php
cms-web_1  | Configuring Backups
cms-web_1  | Configuring Maintenance
cms-web_1  | Running maintenance
cms-web_1  | a:2:{s:5:"value";O:8:"stdClass":2:{s:8:"statuses";a:30:{i:0;O:8:"stdClass":26:{s:10:"created_at";s:30:"Fri Feb 17 08:30:07 +0000 2017";s:2:"id";i:8325074049
69508865;s:6:"id_str";s:18:"832507404969508865";s:4:"text";s:133:"Star Wars: The Last Jedi's Rey Wields A Lightsaber And Has New Hairstyle In Toy Image https://t.co/xNfpvcV0u9
https://t.co/cEOP6IWwSX";s:9:"truncated";b:0;s:8:"entities";O:8:"


I presume it checks the entire DB? 
It always ends on this:
";s:26:"profile_sidebar_fill_color";s:6:"DDEEF6";s:18:"profile_text_color";s:6:"333333";s:28:"profile_use_background_image";b:1;s:20:"has_extended_profile";b:0;s:15:"default_profile";b:1;s:21:"default_profile_image";b:0;s:9:"following";N;s:19:"follow_request_sent";N;s:13:"notifications";N;s:15:"translator_type";s:4:"none";}s:3:"geo";N;s:11:"coordinates";N;s:5:"place";N;s:12:"contributors";N;s:15:"is_quote_status";b:0;s:13:"retweet_count";i:0;s:14:"favorite_count";i:0;s:9:"favorited";b:0;s:9:"retweeted";b:0;s:4:"lang";s:2:"en";}i:48;O:8:"stdClass":24:{s:10:"created_at";s:30:"Thu Apr 27 09:27:02 +0000 2017";s:2:"id";i:857526490287808512;s:6:"id_str";s:18:"857526490287808512";s:4:"text";s:91:"@CCCUNursing @Howest_Intl healthcare in Israel

I can access the MySQL instance. The library files and db files seem ok in the shared dir. Corrupt DB?
Anyone has a clue what’s going on?

2

When you try to access CMS what do you see there?

If you run the down and up commands, does that starts all containers or is the web container not starting?
After the above if you do docker ps -a what does that show?

Is your db really large? perhaps a lot of log/stats files in there? which could explain why maintenance is running so long?

3

Thank you for replying!

docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS                               NAMES
305359368b68        phpmyadmin/phpmyadmin                "/run.sh phpmyadmin"     About an hour ago   Up About an hour    0.0.0.0:1080->80/tcp                phpmyadmin
2d36e3422142        xibosignage/xibo-cms:release_1.8.1   "/entrypoint.sh"         About an hour ago   Up About an hour    127.0.0.1:8080->80/tcp              xibo_cms-web_1
69958d39901c        xibosignage/xibo-xmr:release_1.8.1   "/entrypoint.sh"         About an hour ago   Up About an hour    0.0.0.0:9505->9505/tcp, 50001/tcp   xibo_cms-xmr_1
4aef3675cf6f        mysql:5.6                            "docker-entrypoint..."   About an hour ago   Up About an hour    3306/tcp                            xibo_cms-db_1

in shared/db/cms the log.ibd file is 18G so that can explain a few things.

4

So, all containers seems to be up and running now, 18G log could easily explain why the maintenance was taking it’s time to finish.

That being said even with maintenance running, if all containers are up you should be able to access the CMS, what do you see when you try it?

5

Apache error:
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server

6

This implies that the container is being recreated - i.e. its been pulled freshly from Docker Hub and is starting up for the first time. This may not be a problem, just interesting to note.

The cruft you get output after “Running Maintenance” is your cache being assessed and purged - it shouldn’t emit anything, but it does. You might try:

docker-compose down
rm -R shared/cms/library/cache/*   <<< you might need to be root for that.
docker-compose up

Make sure you’re in the right folder structure before issuing the commands - you don’t want to accidentally recursively delete the wrong things. In this case you want to delete the contents of the cache folder in your library folder.

7

If you have an 18GB log.idb file, then the first thing to do is clear all that out.

Maintenance will do it eventually, but it will struggle with that many records.

Run the following:

docker stop xibo_cms-web_1
docker stop xibo_cms-db_1

That will stop the Xibo CMS and database containers running. Now we’ll just start the database container:

docker start xibo_cms-db_1

Once the database container is running, connect to it as follows:

docker exec -ti xibo_cms-db_1 bash

That will get you a shell inside the MySQL container.

Now open a MySQL command prompt:

mysql -u cms -p cms

Enter the password you chose for the MySQL user in your config.env file.

Now run the following:

TRUNCATE log;

That will delete all entries in the log table.

Now exit from MySQL with the exit; command, and then from the MySQL container with exit.

Now try starting your containers again as normal and see if the CMS is now responsive.

If it is, check the logs and see if there’s something logging alot of data in there quickly that might explain how you’ve ended up with a log table of that size - as in normal operation, a heathy CMS should log almost nothing.

8

Ok the total size of the MySQL cms folder is now 90MB. So that’s ok. But when I restart the containers again it still hangs on running maintenance (I’ve stopped it after 3 hours) but no further messages or errors. The CMS is still not up.
Any ideas?

9

If you want to send your data over today I can take a look.

To do so, stop all the containers, and then zip/tar.gz up the whole installation directory (docker compose files, and shared), and the upload them via the method I will share with you via PM.

I can then run your instance up here and have a look.

10

Issue appears to be an incorrect XMR Address setting - causing a long timeout delay when the maintenance script rebuilding layouts triggers XMR messages to be sent.

11

Thank you for the support. The issue is resolved by updating the XMR setting in the database.

1 Like
12

Dear Alex,

Could you please help me to get the issue fixed? I got same problem as topic starter had.
The installation is absolutely fresh. Version is 1.8.9. After unexpected power-off last day I can see only blank screen when trying to reach CMS interface. It is running maintenance as is written in logs. Log.ibd file reached 9GB. I truncated it on the base of your advice in this thread and it is 114MB now.

docker-compose logs cms-web shows this

Attaching to xibodocker189_cms-web_1
cms-web_1 | Starting cron
cms-web_1 | Starting webserver
cms-web_1 | AH00558: httpd: Could not reliably determine the server’s fully qualified domain name, using 172.17.0.4. Set the ‘ServerName’ directive globally to suppress this message
cms-web_1 | Waiting for MySQL to start - max 300 seconds
cms-web_1 | MySQL started
cms-web_1 | DBVersion
cms-web_1 | 140
cms-web_1 | Existing Database, checking if we need to upgrade it
cms-web_1 | Configuring Maintenance
cms-web_1 | crontab: can’t open ‘/etc/crontabs/apache’: Permission denied
cms-web_1 | Running maintenance
cms-web_1 | Waiting for MySQL to start - max 300 seconds
cms-web_1 | MySQL started
cms-web_1 | DBVersion
cms-web_1 | 140
cms-web_1 | Existing Database, checking if we need to upgrade it
cms-web_1 | Configuring Maintenance
cms-web_1 | crontab: can’t open ‘/etc/crontabs/apache’: Permission denied
cms-web_1 | Running maintenance
cms-web_1 | Waiting for MySQL to start - max 300 seconds
cms-web_1 | MySQL started
cms-web_1 | DBVersion
cms-web_1 | 140
cms-web_1 | Existing Database, checking if we need to upgrade it
cms-web_1 | Configuring Maintenance
cms-web_1 | crontab: can’t open ‘/etc/crontabs/apache’: Permission denied
cms-web_1 | Running maintenance

docker-compose logs cms-db shows this

cms-db_1 | 2018-05-24 07:00:35 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1 | 2018-05-24 07:00:35 0 [Note] mysqld (mysqld 5.6.40) starting as process 1 …
cms-db_1 | 2018-05-24 07:00:35 1 [Note] Plugin ‘FEDERATED’ is disabled.
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Using atomics to ref count buffer pool pages
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: The InnoDB memory heap is disabled
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Memory barrier is not used
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Compressed tables use zlib 1.2.3
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Using Linux native AIO
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Using CPU crc32 instructions
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Initializing buffer pool, size = 128.0M
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Completed initialization of buffer pool
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Highest supported file format is Barracuda.
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: 128 rollback segment(s) are active.
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: Waiting for purge to start
cms-db_1 | 2018-05-24 07:00:35 1 [Note] InnoDB: 5.6.40 started; log sequence number 6236423
cms-db_1 | 2018-05-24 07:00:35 1 [Note] Server hostname (bind-address): ‘*’; port: 3306
cms-db_1 | 2018-05-24 07:00:35 1 [Note] IPv6 is available.
cms-db_1 | 2018-05-24 07:00:35 1 [Note] - ‘::’ resolves to ‘::’;
cms-db_1 | 2018-05-24 07:00:35 1 [Note] Server socket created on IP: ‘::’.
cms-db_1 | 2018-05-24 07:00:35 1 [Warning] Insecure configuration for --pid-file: Location ‘/var/run/mysqld’ in the path is accessible to all OS users. Consider choosing a different directory.
cms-db_1 | 2018-05-24 07:00:35 1 [Warning] ‘proxies_priv’ entry ‘@ root@3c6c194f3377’ ignored in --skip-name-resolve mode.
cms-db_1 | 2018-05-24 07:00:35 1 [Note] Event Scheduler: Loaded 0 events
cms-db_1 | 2018-05-24 07:00:35 1 [Note] mysqld: ready for connections.
cms-db_1 | Version: ‘5.6.40’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 MySQL Community Server (GPL)

Can’t show you a XMR address setting without your help. I changed it when interface was accessible. But I suppose, something has been made incorrectly because I could not receive any screenshots from Windows Player.

When I got same issue last time - I just made a fresh reinstall. Everything was OK until last power down.

Thanks a lot for the great product!

13

If you’re using Docker, then the XMR internal address is set correctly for you out of the box, and then hidden.

The only setting we expose is the external address, and that can’t cause slow startup.

Do you have proof of play stats enabled? If so, perhaps you have a huge backlog of those which it’s busy archiving or deleting?

cms-web_1 | crontab: can’t open ‘/etc/crontabs/apache’: Permission denied

That line is somewhat concerning. Have you tried to modify that file perhaps? If that isn’t setup correctly, then maintenance (XTR) won’t be running normally and that will explain why you have a huge backlog.

14

Hi Alex,

Thanks for your answer!

No, I didn’t modify any of files instead of those listed here https://xibo.org.uk/manual/en/install_docker_linux.html

And I commented first two rows in docker-compose.yml (version and services) because docker had issues with them not allowing bootstrap.

I am not sure about “proof of play stats” because tried to follow manuals.

Just for information if it might be necessary

xibo@xibosrv:/opt/xibo/xibo-docker-1.8.9$ docker -v
Docker version 1.13.1, build 092cba3
xibo@xibosrv:/opt/xibo/xibo-docker-1.8.9$ docker-compose -v
docker-compose version 1.8.0, build unknown

Crontab string concerned me too… Haven’t found solution yet. And there is no folder named “crontabs” though.

15

Can you get me some remote access to it (TeamViewer or SSH) and I can look. DM me the details.

There’s definitely a crontabs folder in the container I have here for 1.8.9, so I suspect yours has been modified in some way.

17

Alex, I have updated docker and docker-compose to the latest versions and made a fresh cms installation with uncommented rows. Let’s see how it goes.

Thanks a lot!

18

You shouldn’t need to start over as a result.

You should be able to use your existing docker-compose.yml, config.env and shared folder.

19

An existing setup continued to show the crontab message in logs and didn’t start. I supposed, it is better to start over with new docker version.
I still have the previous folder and can go back there.

1 Like
20

Hello! I too am getting the following issue:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server

I was logged in to the CMS, editing some images in some regions in a layout. I went down to turn on the display that runs the layout and the Display couldn’t talk to the server to download the updated layout. I went back to my computer to try and access the CMS and was prompted with the above error.

The Linux box that is running the CMS (xibo-docker-1.8.10) is running okay, but Xibo is not.

docker ps -a shows:

CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                     PORTS                                NAMES
284acb3efe05        xibosignage/xibo-cms:release-1.8.10   "/entrypoint.sh"         2 hours ago         Up 4 minutes               127.0.0.1:8080->80/tcp               xibo_cms-web_1
102b49b43754        xibosignage/xibo-xmr:release-0.7      "/entrypoint.sh"         2 hours ago         Up 4 minutes               50001/tcp, 0.0.0.0:65500->9505/tcp   xibo_cms-xmr_1
72778f75132d        mysql:5.6                             "docker-entrypoint..."   2 hours ago         Up 4 minutes               3306/tcp                             xibo_cms-db_1
3b452d31c3ae        hello-world                           "/hello"                 12 months ago       Exited (0) 12 months ago                                        xenodochial_sammet

docker-compose logs cms-web shows

Attaching to xibo_cms-web_1
cms-web_1  | Waiting for MySQL to start - max 300 seconds
cms-web_1  | MySQL started
cms-web_1  | DBVersion
cms-web_1  | 141
cms-web_1  | Existing Database, checking if we need to upgrade it
cms-web_1  | Updating settings.php
cms-web_1  | Configuring Maintenance
cms-web_1  | Removing web/install/index.php from production container
cms-web_1  | Running maintenance
cms-web_1  | Waiting for MySQL to start - max 300 seconds
cms-web_1  | MySQL started
cms-web_1  | DBVersion
cms-web_1  | 141
cms-web_1  | Existing Database, checking if we need to upgrade it
cms-web_1  | Configuring Maintenance
cms-web_1  | crontab: can't open '/etc/crontabs/apache': Permission denied
cms-web_1  | Running maintenance
cms-web_1  | Waiting for MySQL to start - max 300 seconds
cms-web_1  | MySQL started
cms-web_1  | DBVersion
cms-web_1  | 141
cms-web_1  | Existing Database, checking if we need to upgrade it
cms-web_1  | Configuring Maintenance
cms-web_1  | crontab: can't open '/etc/crontabs/apache': Permission denied
cms-web_1  | Running maintenance

docker-compose logs cms-db shows

Attaching to xibo_cms-db_1
cms-db_1   | 2018-07-31 09:43:17 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2018-07-31 09:43:17 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...
cms-db_1   | 2018-07-31 09:53:45 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2018-07-31 09:53:45 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...
cms-db_1   | 2018-07-31 12:04:12 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cms-db_1   | 2018-07-31 12:04:12 0 [Note] mysqld (mysqld 5.6.36) starting as process 1 ...

Any ideas? I can’t work out why it would suddenly just stop working. Thanks.

21

You need to let maintenance run. On Windows particularly, it can take some time to do so.

It can also be a symptom of incorrectly configured XMR settings, although it’s unlikely, especially if it’s an install that has always been on Docker.

If you just wait, it will start correctly given time.