Clicking "Edit" on Library Media file crashes MySQL (1.8rc1)

I have one particular media item in our Library where the MySQL instance (running in Docker) crashes if we click the Edit menu item. A few details:

  • This happens for several different users on the same file
  • It only happens with one particular file
  • If I click on the Media menu item and Edit a different file, then go to Edit the “bad” file, MySQL does NOT crash (but the edit modal does not appear).
  • If I click on the Media menu item and Edit the “bad” file, this is when MySQL crashes.
  • If I click Edit on the “bad” file and click away, I can restart MySQL and continue working.
  • If I click Edit on the “bad” file and do NOT click away, when MySQL restarts the edit modal DOES appear for the “bad” file.

I am quite happy to simply replace this file and see if that works, but I thought, perhaps, you’d like me to collect some logs in case there’s a bug (as opposed to simply “bad data”). Please let me know. And if you do want logs collected, please let me know how - I’m new to Docker and still a little befuddled by it. Thanks!

BTW, if you do NOT want logs, please tell me so I can delete the “Bad” file and move on. Thanks!

Seems rather odd, unless the file itself is corrupted in some way.

It would be good to enable debugging in CMS then try to edit this file and collect logs after that to see what it tries to do and why it fails to do it.

Unfortunately (and I mean this sincerely since I am expert in problem determination, and I do NOT like moving targets!) the problem statement has changed after my work gathering logs.

It is now:

  • Any time I navigate to another screen and then navigate back to “Media” the database server crashes.

This is definitely not what I saw previously, as I was able to navigate to “Media” and Edit other media files.

For some reason, setting the Log Level to “Emergency” doesn’t cause any entries to be made in the Log. I can’t set the Log Level to anything below “Error”. I do have a short log from using the Report Fault wizard.

Thoughts on how to approach this?

You will want to have Debug in log level (debugging from Report fault page does that), perhaps even test server mode to display even more logs, as it seems something is quite wrong with the db now.

OK - couple of things:

  1. I had to set the CMS to Test mode before I could change the Log Level to anything below Error. I set the Log Level to Debug after changing the CMS to Test mode and it worked.

  2. With Debug Log Level I could see a Media item (#56) was the only one specified in the logs after the database crashed and was restarted. So presumably this Media item was the culprit.

I did a Replace on the file (a JPG) and this did not solve the problem.
I deleted the Media item and the problem resolved.
I re-added the same JPG as a new Media item and the problem returned. Re-deleting fixed the problem again.

Clearly there was a problem with the JPG. I’ll post again here if we can figure out what the problem is.

Problem is resolved:

The JPG was only 1.3MB file size. But the resolution was 8000x4500. We reduced that to 2000x1125 and it works perfectly.

I would LOVE to work with you on root cause if you like. You definitely don’t want MySQL servers crashing! It might be quicker/easier to do this with a remote web session, and I’m happy to schedule that with you if you like. I’m guessing it’s the conversion to thumbnail, since we do not have the same problem with Xibo 1.7 (which has no thumbnail) using the same high-res file.

So Xibo doesn’t store the file content in the database at all (just it’s properties - name, size, path etc), so I can’t think why the file would cause a crash. It’s certainly not something we’ve seen before.

Why does MySQL crash? If you look at the container logs, it should give some idea. I’d guess it might be some kind of OOM scenario.

What did the Xibo logs look like on edit with full debugging on? It would have recorded all the SQL statements it was going to run before the crash - at least in theory.

I don’t think it can be thumbnailing because the MySQL server has no role in that - it’s all handled by the cms container.

If you want to make the file available to us I can try it here and see if I can see what the issue is :slight_smile:

Thanks Alex. We had assumed (and hoped) the DB didn’t store the binary file.

It’s entirely possible it’s OOM - we’re using a very small AWS instance for testing. We’ll try to bump the memory and see if that resolves it. Though it seems strange that a 1.3MB file would cause an OOM…

Do you have any guidance on how to access the container logs? I’m new to Docker and started to get acclimated but if you have a suggestion on where to look, that’d be helpful.

We do have the Xibo logs - will make them available if we can’t diagnose on our own. If we are able to figure it out, we’ll post here for public knowledge.

Cheers!

We use the mysql docker image from the MySQL project directly.

The documentation for that container is here:
https://hub.docker.com/_/mysql/

To see the logs of the running container, you’d run:

docker logs xibo-cms-db

Assuming you’re running using our launcher, and used xibo as the instance name - otherwise substitute the container name from docker ps for your database container.

Alternatively, you can get a shell inside the MySQL container by running

docker exec -it xibo-cms-db bash

and then look in /var/log/mysql as normal.

Finally you can destroy and reinitialise your MySQL container to ensure that’s all up to date and as it should be. If you’re using launcher, backup your shared directory (as root as some files will be readable only by that user), then run

./launcher destroy
./launcher bootstrap

That will destroy your CMS and MySQL containers, and rebuild them with your existing data.

I cannot begin to describe how grateful I am for this… I’ve been trying to understand how to access file systems in a Docker container - and I guess I was overcomplicating things. Thank you SO much for the help with this!

No problem. Just keep in mind everything inside the container is considered transient - except if it’s mapped to a volume outside (for example the stuff we put in the shared directory - your MySQL database files in this case). You can make changes to the container, but they’ll be lost when the container is rebuilt (either at upgrade or if it were destroyed in some other way), so in general you want to avoid doing that.