Upgrade to v 1.7.7 -> daily or sometimes hourly client freezes

[irfan]

Dear Dan, thank you for the reply…

@dan

You have the player and the CMS running on the same PC? I didn’t realise that… are you 100% sure that your CMS library and player library locations are different?

No CMS and players are differently installed.
I’ve CMS Server Installed, with players getting the updates from it.

@dan

That is very different from what I understand to be the problem - my understanding was that you received the XLF locked message and the player reverted to the default layout or splash screen. This is the first mention of it actually showing the layout without data? I don’t think Jasmina has the same issue.

7 16-05-25 09:50:48 TRD-SP-SMIT-00 Client [UI Thread] MainForm - ChangeToNextLayout Layout Change to C:\Xibo Library\16.xlf failed. Exception raised was: The process cannot access the file ‘C:\Xibo Library\16.xlf’ because it is being used by another process.

For me this does not go go to the default layout or the splash screen it just does not show contents on the layout instead the layout just shows with the background image endlessly until the player is closed and restarted.

@dan

Allow me a little while to look into that.

Please let me know how you would like to see remotely

Appreciate your assistance on this, as it is just a minor issue, which is causing the employees to to report it to the higher ups that the system is not working, most of the time,
hope you understand.

[dan]

@irfan

No CMS and players are differently installed.

Good…

You should be running maintenance but I do not think it will have any bearing on this issue.

@irfan

For me this does not go go to the default layout or the splash screen it just does not show contents on the layout instead the layout just shows with the background image endlessly until the player is closed and restarted.

This sounds like the “OR” from the first message in the topic:

the campaign will get stuck
on a certain layout, or layout will be missing a region and client will not
switch to the next scheduled layout.

I’ve no explanation for how that results in a half populated layout - if the XLF file cannot be opened then the layout definition cannot be read and the player will not try to load the new layout.

You think it might actually be the prior layout that remains on screen?

[irfan]

@dan

I’ve no explanation for how that results in a half populated layout - if the XLF file cannot be opened then the layout definition cannot be read and the player will not try to load the new layout.

You think it might actually be the prior layout that remains on screen?

that would be possible… that the previous layout is stuck on the screen without the contents, failing to load the next one.

Have you come across this issue, if so this could be one simple answer to this issue, which would be resolving the continous play hanging during the layout change.

thanking you in advance…

[irfan]

Hi Dan,
Do you think it would be stuck on a layout if the update interval is of any region is 120 as default and and get locked…

is there a table or format that we can the update intervals to update the layout

[dan]

Disclaimer: I am not sure this will work!

I’ve created a temporary MSI for you both to install and will PM it to you.

We aren’t allowed to distribute handle.exe, so the MSI assumes that it is present on the machine at c:\handle.exe - so please make sure you download it and put it there before running the MSI.

Xibo will catch the error as before and immediately run handle filtered for the locked file name. It will capture the output and log it to the log window, along with the original message (renamed slightly to IOException to make it clearer).

Can you try the new MSI and post the error you get here (if any)?

[irfan]

Thank you very much Dan,
Highly Appreciated…
Will let you know the results…

[irfan]

Hi dan, Just noticed the below error while editing the update interval on the layout ticker from 120 to 2 minutes,
hope nothing serious.

ob_end_flush(): failed to delete and flush buffer. No buffer to delete or flush
8
Notice
C:\xampp\htdocs\xibo\lib\data\layout.data.class.php
1926

thanks

[dan]

That isn’t a problem (it is expected in some circumstances) and actually happened when you exported a layout.

[fourDegrees]

Just stumbled upon this thread as i had my Nth call today about locked up 1.7.7 clients. I’m late to the party so I don’t have the testing done like irfan or Jasmina but the symptoms I have been having are exactly the same. Was on 1.6.1 and rock solid. Built new CMS on 1.7.7 upgraded the clients and connected them to the new server. Ever since they lock up, totally randomly, sometimes hourly, sometimes daily, sometimes weekly. And they freeze hard. Reboot scripts dont run plus RDP, MS Remote Control, and VNC all will not connect in the locked state so they stay froze until the box is hard reset. Edit: can still connect remotely to the file system to check logs however my new 1.7.7 server has the logging disabled as i am only seeing logs from the 1.6.1 days. Grrr, But this shows that it is not locking more like stalling windows. Some things still work, unfortunately nothing that can be used to kick start the client again does.

Glad to see this thread as I hope you guys can fix this for the next release. Would be happy to test as well now that I am here.

[dan]

Very happy for you to run some tests too, but it does sound to me like you have a different problem. In the OP’s case the word “freeze” is perhaps not the correct word. The player is still running and responsive to user interaction (for example you can open the info window, etc).

It sounds like in your case the UI has hard locked - for RDC/VNC to be locked you’d be looking at something in Windows locking rather than something in the Xibo executable (not saying it isn’t being caused by Xibo).

If my above suspicious are correct, then we should discuss on another topic (just hit reply as linked topic to the right of this message) - my first instinct would be to check the disk usage on the player. Xibo may have eaten it all which caused windows to become unstable.

[Jasmina]

Thanks for your time and effort Dan regarding this issue, I’m surprised there are no more reports of this since I see many people talking about running Intel Compute sticks and such, which are very similar HW to our clients.

Anyhow, to answer Irfan, yes we were (and are) running the maintenance script all along, from cron every 5 minutes; our CMS is Debian based.

We continue to have this happen to several (one or two, each time different) clients approximately once per day, and as suspected running the “result” filter did not catch anything due to the mentioned filter column drop/timing “problem” with the process monitor.

We will definitely implement your modified client with Handle tie-in on a couple of machines but I’d like to wait out the current setup with our test client (it’s bound to crash sooner or later :)). Filtering by [PATH contains XLF] is accumulating a manageable amount of real-time data and clearing the log once or twice a day (assuming no “crash”) is not problematic, and this will definitely catch the offender since we’re logging everything before and after the event (you’re right that saying “crash” or “freeze” may be slightly misleading as to what we’re talking about in this case).

I have no doubt that you know exactly what you’re doing but I’m slightly concerned that calling Handle after the Xibo client has already logged the error may miss the problem event if the lock is indeed some split-second thing?

Also I think we should keep in mind that either Irfans registry tweak and/or our setup of folder options regarding thumbnails have UNDOUBTEDLY reduced the frequency at which the problem occurs.

We’ll report back as soon as there’s any results.

[dan]

@Jasmina

Also I think we should keep in mind that either Irfans registry tweak and/or our setup of folder options regarding thumbnails have UNDOUBTEDLY reduced the frequency at which the problem occurs.

We will definitely be adding that fact to the manual - and if at all possible as a checkbox option in the installer. I’ve added a report so I don’t forget.

@Jasmina

I have no doubt that you know exactly what you’re doing but I’m slightly concerned that calling Handle after the Xibo client has already logged the error may miss the problem event if the lock is indeed some split-second thing?

I was worried about that too - hence the disclaimer - I’ve executed handle as soon after the IOException as I possibly can… if that doesn’t catch it, then it might be as simple as a quick retry loop to open the file.

I appreciate you want to wait for the error to occur with your other test set up - I do actually think it will produce better results, although you obviously have to look at it more frequently to detect it.

@Jasmina

Thanks for your time and effort Dan regarding this issue

And thanks for your patience and your own investigation - it is excellent to have someone (both of you actually), willing to do some serious debugging

[irfan]

Good Morning Dan and Jasmina,

Yep the same error has popped up in my environment, this time with the expected error result as Dan was looking for as below… i hope this would help to get there and resolve the issue… and this is the only issue i’m having which is not allowing me to run the services 24/7 as expected…

one more thing i have noticed this error dosent loggged on the CMS until the xibo player have been restarted…

thanks Dan / Jasmina

9 16-05-26 08:34:16 TRD-SP-SMIT-04 Client [UI Thread] MainForm -
ChangeToNextLayout Layout Change to C:\Xibo Library\32.xlf failed. Exception
raised was: The process cannot access the file 'C:\Xibo Library\32.xlf' because
it is being used by another process.

8 16-05-26 08:34:16 TRD-SP-SMIT-04 Client [UI Thread] MainForm -
ChangeToNextLayout Prepare Layout Failed. Exception raised was: The process
cannot access the file 'C:\Xibo Library\32.xlf' because it is being used by
another process.

7 16-05-26 08:34:16 TRD-SP-SMIT-04 Client [UI Thread] MainForm - PrepareLayout
Handle Output: Handle v4.0Copyright (C) 1997-2014 Mark RussinovichSysinternals -
www.sysinternals.comusage: handle [[-a [-l]] [-u] | [-c <handle> [-y]] | [-s]]
[-p <process>|<pid>] [name] -a Dump all handle information. -l Just show
pagefile-backed section handles. -c Closes the specified handle (interpreted as
a hexadecimal number). You must specify the process by its PID. WARNING: Closing
handles can cause application or system instability. -y Don't prompt for close
handle confirmation. -s Print count of each type of handle open. -u Show the
owning user name when searching for handles. -p Dump handles belonging to
process (partial name accepted). name Search for handles to objects with <name>
(fragment accepted).No arguments will dump all file references.

6 16-05-26 08:34:16 TRD-SP-SMIT-04 Client [UI Thread] MainForm - PrepareLayout
IOException: System.IO.IOException: The process cannot access the file 'C:\Xibo
Library\32.xlf' because it is being used by another process. at
System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at
System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32
rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions
options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy,
Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String
path, FileMode mode, FileAccess access, FileShare share) at
System.IO.File.Open(String path, FileMode mode, FileAccess access, FileShare
share) at XiboClient.MainForm.PrepareLayout(String layoutPath) in
c:\Users\dan\www\xibo-dotnetclient\MainForm.cs:line 553

5 16-05-26 07:59:48 TRD-SP-SMIT-07 Client [UI Thread] MainForm -
ChangeToNextLayout Layout Change to C:\Xibo Library\35.xlf failed. Exception
raised was: The process cannot access the file 'C:\Xibo Library\35.xlf' because
it is being used by another process.

4 16-05-26 07:59:48 TRD-SP-SMIT-07 Client [UI Thread] MainForm -
ChangeToNextLayout Prepare Layout Failed. Exception raised was: The process
cannot access the file 'C:\Xibo Library\35.xlf' because it is being used by
another process.

3 16-05-26 07:59:48 TRD-SP-SMIT-07 Client [UI Thread] MainForm - PrepareLayout
Handle Output: Handle v4.0Copyright (C) 1997-2014 Mark RussinovichSysinternals -
www.sysinternals.comusage: handle [[-a [-l]] [-u] | [-c <handle> [-y]] | [-s]]
[-p <process>|<pid>] [name] -a Dump all handle information. -l Just show
pagefile-backed section handles. -c Closes the specified handle (interpreted as
a hexadecimal number). You must specify the process by its PID. WARNING: Closing
handles can cause application or system instability. -y Don't prompt for close
handle confirmation. -s Print count of each type of handle open. -u Show the
owning user name when searching for handles. -p Dump handles belonging to
process (partial name accepted). name Search for handles to objects with <name>
(fragment accepted).No arguments will dump all file references.

2 16-05-26 07:59:48 TRD-SP-SMIT-07 Client [UI Thread] MainForm - PrepareLayout
IOException: System.IO.IOException: The process cannot access the file 'C:\Xibo
Library\35.xlf' because it is being used by another process. at
System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at
System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32
rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions
options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy,
Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String
path, FileMode mode, FileAccess access, FileShare share) at
System.IO.File.Open(String path, FileMode mode, FileAccess access, FileShare
share) at XiboClient.MainForm.PrepareLayout(String layoutPath) in
c:\Users\dan\www\xibo-dotnetclient\MainForm.cs:line 553

1 16-05-22 08:47:41 TRD-WS-IT-04 Client [LibraryAgent] LibraryAgent - Run
Exception in Run: The process cannot access the file 'C:\Xibo
Library\stats.xml_131081078169868325' because it is being used by another
process. 

[dan]

@irfan

one more thing i have noticed this error dosent loggged on the CMS until the xibo player have been restarted…

That is normal during a crash - the shut down code won’t be called so the logs won’t be sent before the application ends.

Unfortunately the call to handle.exe didn’t work, which is a real shame. It is printing out usage instructions, which implies that it doesn’t understand a command switch that I’ve passed in. My guess is that the file path needs quoting.

I’ve done that and will provide a new XiboClient.exe for you to try again in the PM.

It is also interesting that stats.xml_131081078169868325 was also locked! I am sure there is something else locking your files now

[irfan]

Thanks Dan, for the feedback… will wait for the new exe, just is it ok to install it over the existing one or do i need to uninstall the previous version.
as i noticed that if you install over it in the watchdog folder it leaves x68 folder i believe it was a mistake in v1.7.7

[dan]

It will be a single file that you copy over the top and restart the player.

@irfan

i noticed that if you install over it in the watchdog folder it leaves x68 folder i believe it was a mistake in v1.7.7

No mistake, someone wanted a 64 bit watchdog.

[irfan]

i see… thanks
downloaded the exe… will let you know the result as soon as it happens…

[irfan]

Dear Dan, since past 3 hours i have not yet face the error,

however as soon as i replaced new .exe, one of the players gave the below error, and after that player restart was done, since then no errors encountered…
i’m keeping it under observation…

i hope the below error is useful in finding the error source…

once again thank you.

@ Jasmina, FYI i close the process monitor, as now the new new exe that Dan has uploaded is capable of finding the locked file…

[dan]

Interesting - that would imply that even though Handle was executed immediately on error, it couldn’t find anything locking that file which was previously locked.

Lets see if you get any “hits” next time or if Jasmina finds something with process monitor.

[fourDegrees]

@dan

Very happy for you to run some tests too, but it does sound to me like you have a different problem. In the OP’s case the word “freeze” is perhaps not the correct word. The player is still running and responsive to user interaction (for example you can open the info window, etc).

It sounds like in your case the UI has hard locked - for RDC/VNC to be locked you’d be looking at something in Windows locking rather than something in the Xibo executable (not saying it isn’t being caused by Xibo).

If my above suspicious are correct, then we should discuss on another topic (just hit reply as linked topic to the right of this message) - my first instinct would be to check the disk usage on the player. Xibo may have eaten it all which caused windows to become unstable.

Maybe i went a bit too in depth and gave you the wrong idea. The ui can still be interacted with. Here are the logs from when the player freezes:

UI Thread|5/26/2016 1:27:07 PM|Error|MainForm - ChangeToNextLayout|Prepare Layout Failed. Exception raised was: The process cannot access the file ‘C:\Users\signs\Documents\Xibo Library\38.xlf’ because it is being used by another process.
UI Thread|5/26/2016 1:27:10 PM|Error|MainForm - ChangeToNextLayout|Layout Change to C:\Users\signs\Documents\Xibo Library\38.xlf failed. Exception raised was: The process cannot access the file ‘C:\Users\signs\Documents\Xibo Library\38.xlf’ because it is being used by another process.

So throwing my hat in the ring for the exact same problem.