CMS 2.3.8 - suddenly slow run

Get Help
1

Using 2.3.8 (upgrading since 1.8.x), there are ~130 players connected .
Docker Installation on Ubuntu 14.04

I have full set of Maintenance tasks running periodically:

Xibo_CMS_Tasks
Xibo_CMS_Tasks1449×656 73.8 KB

Server has 5.2GB RAM, and usually there is 700MB free RAM.

Xibo_CMS_TOP
Xibo_CMS_TOP728×574 16.3 KB

the httpd sometime has picks of CPU utilization -= however im not sure this is the problem…

Ay idea what can cause the cms to run so slow?

2

I have also noticed in ‘docker state’ that the xibo_cms-web_1 is on the RAM limit
should i increase ram for the container? how ?

Capture
Capture1140×249 9.98 KB

further investigations. the Xibo_cms-web_1 starts at ~70-125MB memory consumption.
after 5 minute from starting the containers - it suddenly reaches to 98%-100% of 1Gb ram (the limit)

i have disabled all the tasks, and this occurs even without browsing the web interface.
also tried to run the docker without ‘-d’ - but nothing usuaful - when the RAM utilization goes up - nothing is written in console.

any suggestion?
perhaps library tidy ?

3

@alex, can you look into this topic - perhaps you have some insights of this problem ?

4

130 Players is a lot for a single server to handle. Honestly I’d expect you to have issues.

You need to look at what is using that RAM inside the container, and track down your bottleneck.

The tasks need to run for Xibo to work properly so disabling them isn’t a viable option.

5

I have disabled them temporary to check if certain procedure is causing it, however since it does not… i will resume them shortly…

i will log into the container to check up on processes…

6

@alex, in TOP within the container – the httpd is the one that consumes all the available memory
from httpd logs (access/error) i cant find anything useful

the 130 players are there quite steady for the past 6 months at least without any problem until recently

how to find what procedure within xibo cms app is allocating all the ram ?

Capture
Capture880×389 10.2 KB

7

The kinds of things that cause heavy loading are backlogged players (ie with days/months backlog of proof of play statistics to upload), recurring schedules that are badly setup - ie anything with minute/hourly repeats. Short collection intervals. Any of those are likely. The only way to know is to enable the logging and see what those Players are doing.

8

Thanks @alex,
i have cleaned up some of the duplicated schedules.
i don’t have any stat collection enabled nor proof of play enabled in the system

collection interval for all players profile is 5 minutes - now changing to 30… will proceed monitring.
i have some recurring events - they are all set to a daily basis recurrences at most, and i have set the date limit to 2025 now (was far in the future…)

you mean enable audit logs for the players - and see what they are sending to the cms ?

9

Enable auditing on one or two displays in the CMS. That way you can see what is happening and how long it’s taking.

I’d also upgrade to 2.3.9 as a matter of course.

10

thanks @alex,
I have set all players for 30 minutes collection interval - this seems to have made some impact - however not very drastic - the server is at 550-950MB Ram (out of 1GB)
i will continue working on he CMS to check ram utilization - so far its much better.

i have set a player for auditing, and couldn’t find anything useful perhaps you might - here is a 100 logs from the cms

system upgrade for 2.3.9 and library tiday will be done tomorrow after nightly DB backup

ID	Run	Date	Channel	Function	Level	Display	Page	Message
151	83b67bc	3/31/2021 14:03	PLAYER	POST	ERROR	Dynamica - GrandK 75"	XFA:DisplayManager	HeartBeatEvent: XMR unresponsive, issue reconfigure. Last HeartBeat: 6 minutes, 51 seconds and 986 milliseconds
334	aebafeb	3/31/2021 14:03	XMDS	POST	INFO	bar-Ilan Led - Gate 10 Replacement	NotifyStatus	"PDO stats: {
    "default": {
        "select": 4,
        "update": 1
    },
    "log": {
        "insert": 3
    },
    "isolated": {
        "update": 1
    },
    "length": 0.055066823959350586
}."
333	aebafeb	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	NotifyStatus	{"availableSpace":9679081472,"totalSpace":12748505088,"packageName":"uk.org.xibo.client","applicationName":"Xibo","deviceName":"477174127062121","width":1920,"height":1080}
332	aebafeb	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	NotifyStatus	Testing whether this display is auditing. 1617224400 vs 1617188621.
331	aebafeb	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	NotifyStatus	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
330	1867927	3/31/2021 14:03	XMDS	POST	INFO	bar-Ilan Led - Gate 10 Replacement	MediaInventory	"PDO stats: {
    "default": {
        "select": 4
    },
    "log": {
        "insert": 3
    },
    "isolated": {
        "update": 1
    },
    "length": 0.07793998718261719
}."
329	1867927	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	MediaInventory	<files><file type="media" id="5" complete="1" lastChecked="2021-03-31 14:04:05" md5="7fb2977f1880dba96af9eab6ad088807" /><file type="media" id="38" complete="1" lastChecked="2021-03-31 14:04:05" md5="637dfc650fff4976910ea46917734925" /><file type="media" id="21" complete="1" lastChecked="2021-03-31 14:04:05" md5="ec3bb52a00e176a7181d454dffaea219" /><file type="media" id="3" complete="1" lastChecked="2021-03-31 14:04:05" md5="67bb26c11dba6c366834e65f5933aff2" /><file type="media" id="36" complete="1" lastChecked="2021-03-31 14:04:05" md5="ebdb8828051637c91109a329b2138f10" /><file type="media" id="18" complete="1" lastChecked="2021-03-31 14:04:05" md5="d59f5fc84ef7747f27baddc07fe81487" /><file type="media" id="1" complete="1" lastChecked="2021-03-31 14:04:05" md5="3c9137d88a00b1ae0b41ff6a70571615" /><file type="media" id="34" complete="1" lastChecked="2021-03-31 14:04:05" md5="4168dc58a5eb3aa3826d8eae85088804" /><file type="media" id="16" complete="1" lastChecked="2021-03-31 14:04:05" md5="926affd0ae3d3b69e6071b26875638c8" /><file type="media" id="32" complete="1" lastChecked="2021-03-31 14:04:05" md5="49b09eb3738e0a16b488297f2b6ede24" /><file type="media" id="14" complete="1" lastChecked="2021-03-31 14:04:05" md5="2286bb4f44d9ea301131a25c5204ca0a" /><file type="media" id="30" complete="1" lastChecked="2021-03-31 14:04:05" md5="1cd48d78f06d33973d9d761d426e69bf" /><file type="media" id="12" complete="1" lastChecked="2021-03-31 14:04:05" md5="afb33470c582079834acd8b9f979ce1a" /><file type="media" id="28" complete="1" lastChecked="2021-03-31 14:04:05" md5="cd287dcbb4564b57692cfed0e2e58345" /><file type="media" id="10" complete="1" lastChecked="2021-03-31 14:04:05" md5="86aadd0451f9e7dafa957e1e61dd2ed7" /><file type="media" id="26" complete="1" lastChecked="2021-03-31 14:04:05" md5="8c0f6fd19bfa5763a0860b5930a66e14" /><file type="media" id="8" complete="1" lastChecked="2021-03-31 14:04:05" md5="621ba5aaf66e52f3def0cf69807104b9" /><file type="media" id="1608" complete="1" lastChecked="2021-03-31 14:04:05" md5="f6c8efa65711e0cbbc99ba72997ecd0e" /><file type="media" id="24" complete="1" lastChecked="2021-03-31 14:04:05" md5="04425bbdc6243fc6e54bf8984fe50330" /><file type="media" id="6" complete="1" lastChecked="2021-03-31 14:04:05" md5="ffcde6f93e4ac6cb6d4dddad45d344db" /><file type="media" id="39" complete="1" lastChecked="2021-03-31 14:04:05" md5="6e3def73254a72ff0b4ff45c7bba7501" /><file type="media" id="22" complete="1" lastChecked="2021-03-31 14:04:05" md5="145ba08b1886ec06ed80385eb768a050" /><file type="media" id="4" complete="1" lastChecked="2021-03-31 14:04:05" md5="b9d7742384bdf912c51b6a1b5d674f7a" /><file type="media" id="37" complete="1" lastChecked="2021-03-31 14:04:05" md5="d50d13f4bedad3dedd108d4baad60494" /><file type="media" id="19" complete="1" lastChecked="2021-03-31 14:04:05" md5="b456ba26691c4eef6de411e682583e5f" /><file type="media" id="2" complete="1" lastChecked="2021-03-31 14:04:05" md5="212480cf1292db459557814e2d7a571d" /><file type="media" id="35" complete="1" lastChecked="2021-03-31 14:04:05" md5="8c4e4024b6242eecdb754344fcfa8b6c" /><file type="media" id="17" complete="1" lastChecked="2021-03-31 14:04:05" md5="758c582a7fd2f30ec9dd8f91803a9882" /><file type="media" id="33" complete="1" lastChecked="2021-03-31 14:04:05" md5="503ca21fd5014ddefc2cd5159766af32" /><file type="media" id="15" complete="1" lastChecked="2021-03-31 14:04:05" md5="dec9b89b936438760291e401c09f52aa" /><file type="media" id="31" complete="1" lastChecked="2021-03-31 14:04:05" md5="1766972e4d03a6c1056b503c41e94430" /><file type="media" id="13" complete="1" lastChecked="2021-03-31 14:04:05" md5="8d72e0c1775cba38c44bf02aee0c8d83" /><file type="media" id="29" complete="1" lastChecked="2021-03-31 14:04:05" md5="c90a4c420dd010a5e95dedb8927a29e7" /><file type="media" id="11" complete="1" lastChecked="2021-03-31 14:04:05" md5="58963d3a57da7a70ac36d331416746c5" /><file type="media" id="27" complete="1" lastChecked="2021-03-31 14:04:05" md5="fe45297f78b1ab6d8ed9b623d2634530" /><file type="media" id="9" complete="1" lastChecked="2021-03-31 14:04:05" md5="d321fa78bb7190a8ca7e14213ef63203" /><file type="media" id="1609" complete="1" lastChecked="2021-03-31 14:04:05" md5="db15414501ff0a084b61fc1273e73e2f" /><file type="media" id="25" complete="1" lastChecked="2021-03-31 14:04:05" md5="96dff559d6119c163e149280ef1db88c" /><file type="media" id="7" complete="1" lastChecked="2021-03-31 14:04:05" md5="6152d3e99b2c61bb3db365d108337483" /><file type="media" id="40" complete="1" lastChecked="2021-03-31 14:04:05" md5="a4951eb27a8aa834469621674d193f30" /><file type="media" id="23" complete="1" lastChecked="2021-03-31 14:04:05" md5="07f146141537e04ee282a965d8053198" /><file type="media" id="1616" complete="1" lastChecked="2021-03-31 14:04:05" md5="afebca2d92e04707176cd489967f13bf" /><file type="media" id="1347" complete="1" lastChecked="2021-03-31 14:04:05" md5="e2fb0a722b0433db6e54dcdddb51d69a" /><file type="media" id="49" complete="1" lastChecked="2021-03-31 14:04:05" md5="2f3a66671f6f0c7d2eba71e40978f392" /><file type="media" id="1461" complete="1" lastChecked="2021-03-31 14:04:05" md5="3803a22e9eb60aa1585ec19a7aaf13b0" /><file type="media" id="1190" complete="1" lastChecked="2021-03-31 14:04:05" md5="70fb631dde13d01c48a975b0f02dc6f3" /><file type="resource" id="23" complete="1" lastChecked="2021-03-31 14:04:05" md5="null" /><file type="layout" id="7" complete="1" lastChecked="2021-03-31 14:04:05" md5="ff9df1a64160322a7c28fa111f9adc4e" /><file type="resource" id="3793" complete="1" lastChecked="2021-03-31 14:04:05" md5="null" /><file type="layout" id="769" complete="1" lastChecked="2021-03-31 14:04:05" md5="2d0a412b1132ed687c9dc2b094865668" /></files>
328	1867927	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	MediaInventory	Testing whether this display is auditing. 1617224400 vs 1617188620.
327	1867927	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	MediaInventory	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
326	344151c	3/31/2021 14:03	XMDS	POST	INFO	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	"PDO stats: {
    "default": {
        "select": 110,
        "update": 1
    },
    "log": {
        "insert": 100
    },
    "isolated": {
        "update": 1
    },
    "length": 0.17707300186157227
}."
325	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	"<?xml version="1.0"?>
<files generated="2021-03-31 14:03:40" fitlerFrom="2021-03-31 14:00:00" fitlerTo="2021-04-07 14:00:00"><file type="media" id="5" size="4460" md5="7fb2977f1880dba96af9eab6ad088807" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=5" saveAs="xibo-layout-scaler.js" download="http"/><file type="media" id="38" size="655983" md5="637dfc650fff4976910ea46917734925" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=38" saveAs="wi-rain.jpg" download="http"/><file type="media" id="21" size="121200" md5="ec3bb52a00e176a7181d454dffaea219" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=21" saveAs="bootstrap.min.css" download="http"/><file type="media" id="3" size="160251" md5="67bb26c11dba6c366834e65f5933aff2" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=3" saveAs="moment.js" download="http"/><file type="media" id="36" size="738913" md5="ebdb8828051637c91109a329b2138f10" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=36" saveAs="wi-night-clear.jpg" download="http"/><file type="media" id="18" size="4832" md5="d59f5fc84ef7747f27baddc07fe81487" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=18" saveAs="xibo-webpage-render.js" download="http"/><file type="media" id="1" size="95785" md5="3c9137d88a00b1ae0b41ff6a70571615" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=1" saveAs="jquery-1.11.1.min.js" download="http"/><file type="media" id="34" size="597609" md5="4168dc58a5eb3aa3826d8eae85088804" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=34" saveAs="wi-fog.jpg" download="http"/><file type="media" id="16" size="1337459" md5="926affd0ae3d3b69e6071b26875638c8" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=16" saveAs="pdf.worker.js" download="http"/><file type="media" id="32" size="2628732" md5="49b09eb3738e0a16b488297f2b6ede24" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=32" saveAs="wi-day-cloudy.jpg" download="http"/><file type="media" id="14" size="2248" md5="2286bb4f44d9ea301131a25c5204ca0a" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=14" saveAs="jquery.marquee.min.js" download="http"/><file type="media" id="30" size="44720" md5="1cd48d78f06d33973d9d761d426e69bf" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=30" saveAs="weathericons-regular-webfont.woff2" download="http"/><file type="media" id="12" size="19684" md5="afb33470c582079834acd8b9f979ce1a" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=12" saveAs="12.otf" download="http"/><file type="media" id="28" size="67016" md5="cd287dcbb4564b57692cfed0e2e58345" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=28" saveAs="weathericons-regular-webfont.ttf" download="http"/><file type="media" id="10" size="116580" md5="86aadd0451f9e7dafa957e1e61dd2ed7" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=10" saveAs="10.ttf" download="http"/><file type="media" id="26" size="35151" md5="8c0f6fd19bfa5763a0860b5930a66e14" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=26" saveAs="weathericons-regular-webfont.eot" download="http"/><file type="media" id="8" size="30060" md5="621ba5aaf66e52f3def0cf69807104b9" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=8" saveAs="8.otf" download="http"/><file type="media" id="1608" size="159638" md5="f6c8efa65711e0cbbc99ba72997ecd0e" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=1608" saveAs="Chart.min.js" download="http"/><file type="media" id="24" size="23739" md5="04425bbdc6243fc6e54bf8984fe50330" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=24" saveAs="font-awesome.min.css" download="http"/><file type="media" id="6" size="1585" md5="ffcde6f93e4ac6cb6d4dddad45d344db" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=6" saveAs="xibo-dataset-render.js" download="http"/><file type="media" id="39" size="782263" md5="6e3def73254a72ff0b4ff45c7bba7501" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=39" saveAs="wi-snow.jpg" download="http"/><file type="media" id="22" size="40668" md5="145ba08b1886ec06ed80385eb768a050" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=22" saveAs="WeatherIcons-Regular.otf" download="http"/><file type="media" id="4" size="21107" md5="b9d7742384bdf912c51b6a1b5d674f7a" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=4" saveAs="flipclock.min.js" download="http"/><file type="media" id="37" size="220168" md5="d50d13f4bedad3dedd108d4baad60494" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=37" saveAs="wi-night-partly-cloudy.jpg" download="http"/><file type="media" id="19" size="380" md5="b456ba26691c4eef6de411e682583e5f" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=19" saveAs="fonts.css" download="http"/><file type="media" id="2" size="28876" md5="212480cf1292db459557814e2d7a571d" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=2" saveAs="jquery-cycle-2.1.6.min.js" download="http"/><file type="media" id="35" size="827270" md5="8c4e4024b6242eecdb754344fcfa8b6c" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=35" saveAs="wi-hail.jpg" download="http"/><file type="media" id="17" size="18126" md5="758c582a7fd2f30ec9dd8f91803a9882" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=17" saveAs="compatibility.js" download="http"/><file type="media" id="33" size="1445897" md5="503ca21fd5014ddefc2cd5159766af32" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=33" saveAs="wi-day-sunny.jpg" download="http"/><file type="media" id="15" size="333388" md5="dec9b89b936438760291e401c09f52aa" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=15" saveAs="pdf.js" download="http"/><file type="media" id="31" size="379804" md5="1766972e4d03a6c1056b503c41e94430" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=31" saveAs="wi-cloudy.jpg" download="http"/><file type="media" id="13" size="12150" md5="8d72e0c1775cba38c44bf02aee0c8d83" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=13" saveAs="xibo-text-render.js" download="http"/><file type="media" id="29" size="40408" md5="c90a4c420dd010a5e95dedb8927a29e7" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=29" saveAs="weathericons-regular-webfont.woff" download="http"/><file type="media" id="11" size="45884" md5="58963d3a57da7a70ac36d331416746c5" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=11" saveAs="11.ttf" download="http"/><file type="media" id="27" size="127706" md5="fe45297f78b1ab6d8ed9b623d2634530" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=27" saveAs="weathericons-regular-webfont.svg" download="http"/><file type="media" id="9" size="27644" md5="d321fa78bb7190a8ca7e14213ef63203" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=9" saveAs="9.otf" download="http"/><file type="media" id="1609" size="6744" md5="db15414501ff0a084b61fc1273e73e2f" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=1609" saveAs="xibo-countdown-render.js" download="http"/><file type="media" id="25" size="26633" md5="96dff559d6119c163e149280ef1db88c" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=25" saveAs="weather-icons.min.css" download="http"/><file type="media" id="7" size="2988" md5="6152d3e99b2c61bb3db365d108337483" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=7" saveAs="xibo-image-render.js" download="http"/><file type="media" id="40" size="339716" md5="a4951eb27a8aa834469621674d193f30" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=40" saveAs="wi-windy.jpg" download="http"/><file type="media" id="23" size="72259" md5="07f146141537e04ee282a965d8053198" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=23" saveAs="animate.css" download="http"/><file type="media" id="1616" size="2023970" md5="afebca2d92e04707176cd489967f13bf" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=1616" saveAs="1616.wmv" download="http"/><file type="media" id="1347" size="837866" md5="e2fb0a722b0433db6e54dcdddb51d69a" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=1347" saveAs="1347.wmv" download="http"/><file type="media" id="49" size="1272" md5="2f3a66671f6f0c7d2eba71e40978f392" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=49" saveAs="49.png" download="http"/><file type="media" id="1461" size="2250279" md5="3803a22e9eb60aa1585ec19a7aaf13b0" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=1461" saveAs="1461.mp4" download="http"/><file type="media" id="1190" size="646536" md5="70fb631dde13d01c48a975b0f02dc6f3" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=M&itemId=1190" saveAs="1190.jpg" download="http"/><file type="resource" id="23" layoutid="7" regionid="17" mediaid="23" updated="1605418138"/><file type="layout" id="7" size="1033" md5="ff9df1a64160322a7c28fa111f9adc4e" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=L&itemId=7" saveAs="7.xlf" download="http"/><file type="resource" id="3793" layoutid="769" regionid="1313" mediaid="3793" updated="1615445659"/><file type="layout" id="769" size="2241" md5="2d0a412b1132ed687c9dc2b094865668" path="https://cms.creativelabs.co.il/xmds.php?file=64096146edff268eeec89173e70e52f89d5e3315ab3ce827188244b153018c09&displayId=32&type=L&itemId=769" saveAs="769.xlf" download="http"/><file type="blacklist"/></files>
"
324	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Testing whether this display is auditing. 1617224400 vs 1617188620.
323	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Creating clock out of possible ["Xibo\\Widget\\Clock"]
322	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	MD5 for layoutid 769 is: [2d0a412b1132ed687c9dc2b094865668]
321	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Testing whether this display is auditing. 1617224400 vs 1617188620.
320	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Loaded 769 with hash c6c8f6313fca87307beb58d3b9665214, status 1
319	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
318	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
317	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Playlist with {"loadPermissions":true,"loadWidgets":true,"loadTags":true,"loadPlaylists":true,"loadCampaigns":true}
316	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
315	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Region with {"loadPlaylists":true,"loadTags":true,"loadPermissions":true,"loadCampaigns":true}
314	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
313	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Playlist with {"loadPermissions":true,"loadWidgets":true,"loadTags":true,"loadPlaylists":true,"loadCampaigns":true}
312	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
311	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Region with {"loadPlaylists":true,"loadTags":true,"loadPermissions":true,"loadCampaigns":true}
310	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
309	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Playlist with {"loadPermissions":true,"loadWidgets":true,"loadTags":true,"loadPlaylists":true,"loadCampaigns":true}
308	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
307	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Region with {"loadPlaylists":true,"loadTags":true,"loadPermissions":true,"loadCampaigns":true}
306	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Loading Layout 769 with options {"loadPlaylists":true,"loadTags":true,"loadPermissions":true,"loadCampaigns":true}
305	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
304	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Creating text out of possible ["Xibo\\Widget\\Text"]
303	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	MD5 for layoutid 7 is: [ff9df1a64160322a7c28fa111f9adc4e]
302	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Testing whether this display is auditing. 1617224400 vs 1617188620.
301	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Loaded 7 with hash 6903a5ae085f5132bb8319eea6a65499, status 1
300	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
299	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
298	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Playlist with {"loadPermissions":true,"loadWidgets":true,"loadTags":true,"loadPlaylists":true,"loadCampaigns":true}
297	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
296	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Region with {"loadPlaylists":true,"loadTags":true,"loadPermissions":true,"loadCampaigns":true}
295	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
294	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Playlist with {"loadPermissions":true,"loadWidgets":true,"loadTags":true,"loadPlaylists":true,"loadCampaigns":true}
293	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
292	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Load Region with {"loadPlaylists":true,"loadTags":true,"loadPermissions":true,"loadCampaigns":true}
291	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Loading Layout 7 with options {"loadPlaylists":true,"loadTags":true,"loadPermissions":true,"loadCampaigns":true}
290	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
289	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Testing whether this display is auditing. 1617224400 vs 1617188620.
288	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
287	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	"Config Combined is: [
    {
        "name": "emailAddress",
        "default": null,
        "value": "bar-ilan@creativelabs.co.il",
        "type": "string"
    },
    {
        "name": "settingsPassword",
        "default": null
    },
    {
        "name": "collectInterval",
        "default": 300,
        "value": 1800,
        "type": "int"
    },
    {
        "name": "downloadStartWindow",
        "default": "00:00"
    },
    {
        "name": "downloadEndWindow",
        "default": "00:00"
    },
    {
        "name": "xmrNetworkAddress",
        "default": null
    },
    {
        "name": "statsEnabled",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "aggregationLevel",
        "default": "Individual",
        "type": "string"
    },
    {
        "name": "orientation",
        "default": 0
    },
    {
        "name": "screenDimensions",
        "default": null,
        "value": "0,0,1920,1080",
        "type": "string"
    },
    {
        "name": "blacklistVideo",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "storeHtmlOnInternal",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "useSurfaceVideoView",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "logLevel",
        "default": "error"
    },
    {
        "name": "versionMediaId",
        "default": null
    },
    {
        "name": "startOnBoot",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "actionBarMode",
        "default": 1
    },
    {
        "name": "actionBarDisplayDuration",
        "default": 30
    },
    {
        "name": "actionBarIntent",
        "default": null
    },
    {
        "name": "autoRestart",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "startOnBootDelay",
        "default": 60
    },
    {
        "name": "sendCurrentLayoutAsStatusUpdate",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "screenShotRequestInterval",
        "default": 0
    },
    {
        "name": "expireModifiedLayouts",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "screenShotIntent",
        "default": null
    },
    {
        "name": "screenShotSize",
        "default": 800
    },
    {
        "name": "updateStartWindow",
        "default": "00:00"
    },
    {
        "name": "updateEndWindow",
        "default": "00:00"
    },
    {
        "name": "dayPartId",
        "default": null
    },
    {
        "name": "webViewPluginState",
        "default": "DEMAND"
    },
    {
        "name": "hardwareAccelerateWebViewMode",
        "default": "2"
    },
    {
        "name": "timeSyncFromCms",
        "default": 0
    },
    {
        "name": "webCacheEnabled",
        "default": 0
    },
    {
        "name": "serverPort",
        "default": 9696
    },
    {
        "name": "installWithLoadedLinkLibraries",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "forceHttps",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "useMultipleVideoDecoders",
        "default": "default",
        "type": "string"
    },
    {
        "name": "maxRegionCount",
        "default": 0
    }
]"
286	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	"Config loaded: [
    {
        "name": "emailAddress",
        "value": "bar-ilan@creativelabs.co.il",
        "type": "string"
    },
    {
        "name": "collectInterval",
        "value": 1800,
        "type": "int"
    },
    {
        "name": "screenDimensions",
        "value": "0,0,1920,1080",
        "type": "string"
    }
]"
285	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	"Config Default is: [
    {
        "name": "emailAddress",
        "default": null
    },
    {
        "name": "settingsPassword",
        "default": null
    },
    {
        "name": "collectInterval",
        "default": 300
    },
    {
        "name": "downloadStartWindow",
        "default": "00:00"
    },
    {
        "name": "downloadEndWindow",
        "default": "00:00"
    },
    {
        "name": "xmrNetworkAddress",
        "default": null
    },
    {
        "name": "statsEnabled",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "aggregationLevel",
        "default": "Individual",
        "type": "string"
    },
    {
        "name": "orientation",
        "default": 0
    },
    {
        "name": "screenDimensions",
        "default": null
    },
    {
        "name": "blacklistVideo",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "storeHtmlOnInternal",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "useSurfaceVideoView",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "logLevel",
        "default": "error"
    },
    {
        "name": "versionMediaId",
        "default": null
    },
    {
        "name": "startOnBoot",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "actionBarMode",
        "default": 1
    },
    {
        "name": "actionBarDisplayDuration",
        "default": 30
    },
    {
        "name": "actionBarIntent",
        "default": null
    },
    {
        "name": "autoRestart",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "startOnBootDelay",
        "default": 60
    },
    {
        "name": "sendCurrentLayoutAsStatusUpdate",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "screenShotRequestInterval",
        "default": 0
    },
    {
        "name": "expireModifiedLayouts",
        "default": 0,
        "type": "checkbox"
    },
    {
        "name": "screenShotIntent",
        "default": null
    },
    {
        "name": "screenShotSize",
        "default": 800
    },
    {
        "name": "updateStartWindow",
        "default": "00:00"
    },
    {
        "name": "updateEndWindow",
        "default": "00:00"
    },
    {
        "name": "dayPartId",
        "default": null
    },
    {
        "name": "webViewPluginState",
        "default": "DEMAND"
    },
    {
        "name": "hardwareAccelerateWebViewMode",
        "default": "2"
    },
    {
        "name": "timeSyncFromCms",
        "default": 0
    },
    {
        "name": "webCacheEnabled",
        "default": 0
    },
    {
        "name": "serverPort",
        "default": 9696
    },
    {
        "name": "installWithLoadedLinkLibraries",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "forceHttps",
        "default": 1,
        "type": "checkbox"
    },
    {
        "name": "useMultipleVideoDecoders",
        "default": "default",
        "type": "string"
    },
    {
        "name": "maxRegionCount",
        "default": 0
    }
]"
284	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10 Replacement	RequiredFiles	Checking permissions against the logged in user: ID: 1, Name: system, UserType: 1
283	344151c	3/31/2021 14:03	XMDS	POST	DEBUG	bar-Ilan Led - Gate 10
11

So if reducing the collection interval helps, it’s something that is being done at collection - so that might be that you have a dataset that’s misconfigured, or an RSS feed or something like that.

Broadly if it’s been fine, and Xibo hasn’t changed (ie you hadn’t just upgraded it), then something scheduled in Xibo, or something external changed. Track that down, and you have your answer. Having detailed graphs of your server load and MySQL loadings etc are a great help there as you can see what changed when the problem started.

12

So, i have upgraded to 2.3.9,
increased cms_Web_1 to 2Gib Ram,
performed massive cleanup, and modified all Players for 30 min collection.

there are peaks of 1.2Gib Ram, however these are cleaned up eventually and cms maintain lower memory utilization.
thanks for the help!

13

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.