How to automatically restart XiboClient when it freezes

ken · April 30, 2020, 11:04am

Hello.

I think that the Xibo Client may freeze due to insufficient capacity of the PC or long operation of the PC.

(In my case, it was the latter. There were subplaylists and datasettickers on the screen, but they were all frozen. The operating system was normal. I looked at the log for that display on the CMS and there were no errors.)

As a countermeasure for this case, I am currently looking for the method in the title.

My current plan is to use PowerShell in windows.
If the CPU time of the XiboClient does not change for a certain period of time, it is assumed to be frozen and the XiboClient is restarted.

However, I wish I could do this with the standard WatchDog without using such a complicated method.
https://xibo.org.uk/docs/setup/windows-player-watchdog
(However, it seems that watchdog checks for the existence of the XiboClient process itself, so I think this is difficult.)

Or, if there are any points in the above method, please let me know.

Environment
CMS 2.2.1
Client v2 -201 (windows)

I look forward to your comments.

alex · April 30, 2020, 1:16pm

The Watchdog already does some quite complex checks on the Player (memory usage, whether certain key threads are running inside the Player), so the question is why it didn’t detect the Player as hung in your case.

When it’s in that state, first thing to do is open the Player status screen (press i) and that may give some indication of what the Player state is.

I would also suggest upgrading your Player to 2R253 (released yesterday) before you do anything else.

ken · May 13, 2020, 5:54am

Hi, Alex.
Thank you for your reply.
Since then I’ve looked at all the settings for watchdog (including those not described in the documentation).
As you said, Alex, I certainly confirmed that players have already taken quite advanced measures.
(The Process.Responding method seemed to be used to detect hangs.)
However, the hang in this case could not be detected by this detection method.

I was able to check the player’s status log from the CMS.
Only the “No active Process” log remains after manually restarting the player application after the hang.
No information about the hang itself was available.
So, as Alex says, “Why did the player hang but not detect it this time?” is the issue.

I have no plans to take advantage of the new version 2R 253, but I would like to test it if I can make some progress on this issue.

Based on the above, my questions are as follows.

・This happened during a long running test of the player (About a month), is it mandatory to reboot regularly?

・The player application after 2R 202 seems to be able to log the state of threads. Are there any plans to consider the function to perform some actions based on this?

I am thinking of various ways to detect hangs that don’t appear in the status log like this, such as CPU time check, but I am in the middle of trial and error due to my lack of technical ability.

I would appreciate it if you could comment.

ken · May 26, 2020, 5:33am

Hello.

I put this issue on hold, but it’s always in the back of my mind. I hope the topic will not be closed.
For verification, I would like to have the client freeze intentionally, what kind of method is there?

As mentioned above, the operating system was normal.
I want to freeze only the application.
By the way, I also have experience with C #, PHP, etc. used for xibo, so there is no problem with source level work.

I want to contribute to the stable operation of Xibo, but I don’t know how to verify it.

dan · June 1, 2020, 10:20am

As Alex said before, in a hang situation we really need to see if the Player is responding to input directly on the device. It might be that the process is perfectly responsive and some logic in the code has hung. Seeing if the status screen opens and what is contained on it will help with this.

If the CMS is still updating for the Display it implies that not all of the Player has stopped working as the Threads which communicate with XMDS must still be functioning.

As with any PC it is good practice to have scheduled downtime for maintenance, windows updates, etc. It would certainly do no harm to have a weekly reboot of the device during off peak hours. The Player itself doesn’t require rebooting.

ken · June 3, 2020, 1:49pm

Hi Dan.
Thank you for your reply.

The same thing happened just yesterday.
Unfortunately, I cannot open the status screen in this situation.

When you try to do so, windows will tell you “Not Responding” and then the player’s process will fall.
Therefore, at the time of the event, as you say, the process is responding, so it is not killed, but some of the processes are thought to be hung. When I try to open the status screen, Windows thinks the player’s main process is dead.

For the time being, I will try to intentionally hang it by incorporating a high-load process such as an infinite loop.
If I found anything, I will comment again.
Thank you.

ken · June 22, 2020, 9:06am

Note： similar topics (Closed)

ken · June 25, 2020, 12:04am

The event happened again a while ago, so I will record it.
I use the most up-to-date environment (cms: 2.3.4 & windows _ client: 254.1)

After I see the player hang, the moment I use the mouse or keyboard to try to open the information screen, the player drops.

I feel like I’m chasing a fleeing mirage when I try to catch it.

ken · June 25, 2020, 2:36am

Unlike v 201, a player log of the time of occurrence was recorded for this event.
The latest version is great!
I haven’t seen it in detail, but there were a few errors related to WMP. (I am Japanese, so the foreign language in the log is Japanese.)

error log：


|ID|実行|日付|チャネル|ファンクション|レベル|ディスプレイ|ページ|メッセージ|
|---|---|---|---|---|---|---|---|---|
|362215|409a7c9|2020/6/25 9:46|XMDS|POST|ERROR|MIC-PSC1|NotifyStatus|SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction|
|362191|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133980|
|362190|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133979|
|362189|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133978|
|362188|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133905|
|362187|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133904|
|362186|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133903|
|362185|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133902|
|362184|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133901|
|362183|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133900|
|362182|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133977|
|362181|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133976|
|362180|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133975|
|362179|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133962|
|362178|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133961|
|362177|ca9b088|2020/6/25 9:00|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133960|
|362170|1f16f22|2020/6/25 8:46|XMDS|POST|ERROR|MIC-PSC1|NotifyStatus|SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction|
|362145|6920664|2020/6/25 8:13|PLAYER|POST|ERROR|MIC-PSC1|[Watcher] Watchdog|No active processes|
|362144|4d35908|2020/6/25 8:12|PLAYER|POST|ERROR|MIC-PSC1|[Watcher] Watchdog|No active processes|
|362143|38f915e|2020/6/25 8:12|PLAYER|POST|ERROR|MIC-PSC1|Main|Unhandled Exception: プログラムの実行を続行するための十分なメモリがありませんでした。|
|362142|38f915e|2020/6/25 8:10|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11C6|
|362141|38f915e|2020/6/25 8:07|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11C6|
|362140|38f915e|2020/6/25 8:03|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11C6|
|362116|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133959|
|362115|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133958|
|362114|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133957|
|362113|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133956|
|362112|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133955|
|362111|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133954|
|362110|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133953|
|362109|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133952|
|362108|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133951|
|362107|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133890|
|362106|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133889|
|362105|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133888|
|362104|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133950|
|362103|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133949|
|362102|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133948|
|362101|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133893|
|362100|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133892|
|362099|a3b50a8|2020/6/25 8:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133891|
|362139|38f915e|2020/6/25 8:00|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11B1|
|362138|38f915e|2020/6/25 7:57|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11B1|
|362137|38f915e|2020/6/25 7:56|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11C6|
|362136|38f915e|2020/6/25 7:53|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11C6|
|362135|38f915e|2020/6/25 7:50|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11C6|
|362134|38f915e|2020/6/25 7:49|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D10BD|
|362133|38f915e|2020/6/25 7:46|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11B1|
|362092|d4c381f|2020/6/25 7:44|XMDS|POST|ERROR|MIC-PSC1|RequiredFiles|SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction|
|362132|38f915e|2020/6/25 7:44|PLAYER|POST|ERROR|MIC-PSC1|[RequiredFilesAgentThread] RequiredFilesAgent - R|Exception in Run: クライアントは 'text/plain;charset=UTF-8' の応答のコンテンツ タイプを見つけましたが、'text/xml' が必要です。要求は以下のエラーにより失敗しました。--There has been an unknown error with XMDS, it has been logged. Please contact your administrator.--.|
|362131|38f915e|2020/6/25 7:43|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D10BD|
|362130|38f915e|2020/6/25 7:40|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D10BD|
|362129|38f915e|2020/6/25 7:36|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D10BD|
|362128|38f915e|2020/6/25 7:33|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = メディア ファイルが見つかりません。|
|362127|38f915e|2020/6/25 7:30|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D10BD|
|362126|38f915e|2020/6/25 7:29|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D10BD|
|362125|38f915e|2020/6/25 7:26|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = メディア ファイルが見つかりません。|
|362124|38f915e|2020/6/25 7:16|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11B1|
|362070|ffd2c93|2020/6/25 7:04|XMDS|POST|ERROR|MIC-PSC1|NotifyStatus|SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction|
|362062|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133947|
|362061|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133946|
|362060|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133945|
|362059|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133923|
|362058|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133922|
|362057|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133921|
|362056|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133920|
|362055|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133919|
|362054|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133918|
|362053|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133917|
|362052|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133916|
|362051|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133915|
|362050|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133456|
|362049|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133455|
|362048|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133454|
|362047|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133453|
|362046|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133452|
|362045|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133451|
|362044|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133450|
|362043|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133449|
|362042|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133448|
|362041|cc07473|2020/6/25 7:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133447|
|362123|38f915e|2020/6/25 7:02|PLAYER|POST|ERROR|MIC-PSC1|Video|MediaElement_MediaFailed: Media Failed. E = HRESULT からの例外:0xC00D11B1|
|361999|6a644d5|2020/6/25 6:06|XMDS|POST|ERROR|MIC-PSC1|NotifyStatus|SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction|
|361994|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133914|
|361993|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133913|
|361992|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133912|
|361991|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133899|
|361990|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133898|
|361989|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133897|
|361988|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133896|
|361987|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133895|
|361986|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133894|
|361985|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133887|
|361984|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133886|
|361983|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133885|
|361982|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133839|
|361981|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133838|
|361980|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133837|
|361979|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133836|
|361978|1f5f2ba|2020/6/25 6:02|XMDS|POST|ERROR|MIC-PSC1|SubmitStats|Stat for a widgetId that doesnt exist: 133835|

I’m beginning to see the direction of this issue, so I’ll write it down again.

・The player freezes.
・It occurs once a month in very rare cases.
・The operating system does not freeze.
・The player terminal is OS: windows 10, CPU: Celeron ・ Memory: 4GB
・The player screen consists of three regions.
・Each region has only one widget.
・There are three widgets: a sub-playlist, an image, and a dataset ticker.
・Images and videos are assigned to the sub-playlist in a dynamic layout.
Images and videos are about 1GB in total, of which about 700 GB is video.
・When a freeze occurs, the sub-playlist portion is completely dark.
・watchdog cannot detect this freeze.
・This is because when you check the status on the CMS screen, the last update time is successfully updated and the display status is up to date.
・The moment you try to open the information screen, the player drops. The process dies.
・The watchdog detects this and reboots immediately.
・Versioning from v 201 to v 254 provided a detailed error log at this time. WMP is causing some errors（0xC00D11B1,0xC00D10BD,0xC00D11C6）.

ken · September 3, 2020, 5:25am

Hello.

While checking the latest source commit, I found a fix for the memory leak reported above.
Thank you very much.

dan · September 3, 2020, 8:53pm

Thanks for coming back to this and marking a solution. Hopefully the referenced issue solved the problem.

Best wishes,
Dan

system · December 4, 2020, 2:53am

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.