Hopefully I’ve got my terminology correct.
I’ve xibo windows client with 3 regions , in regions 1 & 2 I’ve a video feed using a vlc plugin and in the third region I’ve a simple clock widget. Every now and again the video’s will freeze and the clock will also freeze, the client however continues to collect from the xibo server and the watchdog doesn’t notice that the client has hung. If I eventually notice this , it can be several hours and I click on the screen or press a button “i” for instance I’ll get the message from windows to the effect that the xibo client has hung, what do I want to do? If I leave it at this point, the watchdog will correctly notice that the client has hung and restart it.
Any ideas what I can adjust in the watchdog settings to correctly identify when the client has got in this semi-hung state ?
As a temporary workaround I’ve attempted to schedule every 10 minutes a batch file to kill the client so the watch dog restarts it , but that hasn’t been successful either. I’m probably killing the wrong process or using the wrong options.
I’d Ideally like to dispense with windows client completely but that probably won’t be possible until the ubuntu client becomes available.
Any suggestions appreciated as to what to tweek to get the watchdog to pick up the odd behaviour and restart.
Which CMS and player versions are you using please?
Do see any error logs from the player in CMS? (perhaps enable auditing on the player to collect more logs).
Or perhaps anything in the Windows Event Viewer?
Perhaps also have a look at Windows Task Manager for the CPU, Memory etc usage when player is running.
CMS version 1.8.2 - The specific client I’m debugging is running 1.8.2-129 however I’ve other clients running similar layouts with similar problems. running a mix of 1.8.1-128 and 1.8.2-129.
I tried turning on auditing against the specific client but couldn’t figure out where I’m supposed to look for the output of the logging/auditing.
So suspect I’m going to need specific links in the wiki/instructions to achieve this.
Nothing of worth logged in event viewer, not sure how to view task manager as player will always play on top, obscuring the view, also any attempt to switch away from the client results in the usual “Xibo Open Source Digital signage is not responding - close the program / wait for the program to respond question” which of course the watch dog then detects.- message worded differently in original post “what do I want to do?” - sorry couldn’t remember the wording off the top of my head earlier - having a senior moment.
It would be good to upgrade CMS and players to the newest version, just in case there is something bug that you’re encountering there.
Regarding logs sent by the player to the CMS, there is a Log page in CMS (menu on the left under Advanced section). The log page can be filtered by particular display, you can also change the interval/duration back as well.
Xibo for Windows is indeed designed to be always on top, however you can press i on your keyboard which will open player status window and will allow you to change focus to other applications/windows.
I’d not expect to see that error when you try to switch away from Xibo player, unless the PC is under heavy load and/or player has problems showing the content.
I wonder if 2 videos at the same time (which I assume you have looking at your first post) are a bit too much for this PC - could be good to test it with just one video stream and see if the issue will still occur.
Thanks for getting back to me. Sorry for the overly long reply.
I suspected you would say upgrade, when we first installed these particular displays 1.8.2 was the latest version, I’ll upgrade as requested and report back however I doubt it will make a difference, I believe vlc plugins showing rtsp video streams under a windows client is an unsupported feature, so I doubt there’s been any changes to it’s support, hence my desire to figure out why the watchdog isn’t detecting the hang.
The particular pc I’m using as my guinea pig to get to the bottom of this , can run for several days without issue and at other times for 10 minutes, as do they all. I suspect it’s something other than just the 2 video streams that are causing the issue, windows update, virus scan perhaps. pressing “i” causes the “Xibo Open Source Digital signage is not responding - close the program / wait for the program to respond question” message so that’s changing the situation both as far as what is viewable in the task manager and what the watchdog is seeing/noticing.
I have other pc’s showing 4 simultaneous video feeds. They all will lock up at some point during the week with similar symptoms, they don’t all lock up together. I now have around 6 displays showing this type of content. Yes ideally it would be useful to discover why they hang in the first place, however I suspect even if I do as it’s an unsupported method, my only recourse is to get the watchdog to notice the hang.
I’m not seeing any logs under advanced logs so suspect I’ve not setup auditing/logging correctly, the wiki at the time didn’t quite match what I see on my cms to setup logging, I could be looking in the wrong place in the wiki. If you are able to point me to the correct place that would be good.
I’m prepared to purchase the Android hardware if I could be certain it would work, as ultimately they are a neater device, which I may just do anyway, I take it there is no trial facility for hardware?
On Windows it would be better to use hls streams in HLS widget indeed if possible.
RTSP is generally supported only on Android, Xibo for Android comes with 14 days trial licence, as for android hardware, I’m afraid we do not sell it, as such it may depend on the manufacturer.
You could try asking DSDevices and see if they can help you.
the sources are all onvif h.264/h.265 security cameras , so the HLS option isn’t really available unless I start adding in extra hardware just to act as a re-encoder. Which seems a waste especially as the solution for the most part works.
Client ran for 16 hours without issue then at around 11:24am - bang screen frozen - I was informed by a human on this occasion around 3pm, yes they’re not really bothering much down there at the moment, I then pressed “i” , which triggers “Client not responding” - the watchdog then restarted it, it’s then run for another 5 hours or thereabouts, froze at 8:27pm. The audit log stored locally on the client pc shows activity at around the times of the freeze but nothing dissimilar from other previous activity which has passed without hanging the screen.
Apparently you can do a view of processes of a remote pc, using rpc (Microsoft sysinternals/ Lizards remote task manager, to name a few) , so I’ve attempted to set that up to see what if anything the pc thinks it’s doing just before I press “i”. Not had a great deal of luck seems I need to open a lot of ports on the client to get that working properly.
The configuration file for the watchdog now has more options than listed in the documentation here, one of the advantages or benefits of getting leading edge open source software. Unsure whether tweeking any of the extra entries would give me a workaround but open to suggestions.
As a quick workaround I currently have the other clients rebooting once an hour.
Yes I’m looking into the DSDevices option. Also how far along is the Ubuntu client, don’t even mind paying for it assuming it’s not overtly expensive to aid development.
just reading through the specification of the cross platfrom player here I knew there was a lot of work in xibo, but not quite how much, so I’d like to thank you/all the developers who have spent considerable time, effort etc into writting such a worthwhile solution.
hmm if it crashes after you press i that either it runs out of memory or the CPU/GPU usage is at 100% (or both).
Could be good to monitor these resources.
Since it does crashes, it could be good to look at the Microsoft Event Viewer, it may contain additional information about the problem.
I appreciate that troubleshooting this might not be ideal, since you’re obviously aiming for reasonable playback, but it could be good to perhaps set up a test device for troubleshooting purposes, while other devices are set to restart as you have now.
Regarding Linux player, we had a technology preview not long ago, but it will be quite some time for it to be fully functional player.
I’ll not be on site until next Tuesday, However what I’ll do when I do get back is put a second screen on this guinea pig pc, hopefully task manager will run happily on there. Not sure why I didn’t think of that before, so it’s visible with xibo running. I’ve tried remotely monitoring resources, but got emboiled in rpc and wmi permissions and wasn’t succesful at getting connected. I may even be able to point a video camera at that second screen so I can rewind the recording and view what is/was happening when it froze.
To re-cap I’ve plugged in a second monitor and have task manager running on it so we can see what is happening both when the main screen is working and when the main screen has frozen.
When running correctly “xibo open source digital signage” process has the following stats
cpu 5% , Memory 103.6MB disk 0.1MB/sec Network 2.2Mbps
When screen has frozen
Task manager reports
xibo open source digital signage - Not Responding - cpu 0% , Memory 44.8MB, Disk 0.1Mb/sec, Network 0Mbps
No notification from windows that main screen has hung. However although task manager says “not responding” - every now and again the status changes to cpu 0.2%, Memory 44.8MB, Disk 0.1%, Network 0%, and strangely during this state it still manages to connect into the xibo cms and ask for updates. Incidentally non are programmed.
The stats for the complete machine in either state stay pretty constant at CPU 32%, 49 % Memory , 2% Disk, 2% Network
Not sure that this particularly helps diagnose or nudges us in any particular direction for a workaround but open to suggestions.
Ran into this topic while I was searching for information on a similar issue. I’ve got 2 displays with an Intel computestick running the xibo player (windows 10). Both displays were working smoothly until about early July or so. After that my other screen has been having somewhat similar issues than in the opening post. In my case the screen has a jpg “slideshow”, a video section and a still jpg on bottom. Problems start always on the video section first, it suddenly starts showing a “heart rate monitor” graph instead of the video. At some point the slidwshow stops and shows just blank and sometimes even the bottom still jpg is replaced by blank.
Of course I updated the player (about two weeks ago), but to no avail. I’m suspecting the issue to be in some Windows update because my other screen is working just fine and that windows wasn’t updated (both computers had Windows Update- service disabled but for some reason the other windows still had downloaded the big updates).
So I’m not getting the “not responding” message, but otherwise bit similar issues. I could of course make the player to restart every hour or so, but I’m still hoping that someone finds or has found a better solution.