Upgrade to v 1.7.7 -> daily or sometimes hourly client freezes

Get Help
42

The good news: we got tired of waiting for test to crash, so we ran the process monitor on 6 more production clients, and we (finally) caught it!

The bad news: as far as we can tell, it seems to be the Xibo client?

The only thing process monitor (running the “PATH contains XLF” filter) caught was indeed a “SHARING VIOLATION” and no other process is accessing XLF files at any time before or after the event.

We’ve trimmed and exported the logs for the time period around the crash and will PM you the file in .PML format (that can be opened and explored with the Sysinternals process monitor itself). If you’d like the .CSV or .XML please let us know (XML is really large though).

43

Dear All

A Very Good Morning to u all…

Dan, after running the new exe for over 24 hrs, below is the log from one of the players, and one of the player constantly sending the memory error all the night,

hope this error description fro the handle.exe is worth it…

Pasted image1039×676 39.5 KB

44

OK great, thanks. I’ll add you to the PM with the MSI when I have a new one to try.

Yes, this does look like bad news - you didn’t mention the actual time of the locked file message? I assume it coincides with the sharing violation at 6:45:21?

Nope, sorry. I think the handle experiment can be considered a failure - handle is saying that there isn’t anything locking the file immediately after the message occurs, which must mean that the lock time is very short indeed.

I just want to go back to first principles - when we were first discussing this issue you both indicated that you would get this error message repeating over and over, leading to the player never showing the next layout. (maybe this was my assumption)

When Xibo fails to load a layout, it will try the next one in the schedule. If there isn’t another one it will try the default layout, if that isn’t available it will load the splash screen. Once it has a layout loaded and then that expires, it will try again with the layouts in its expected schedule.

Hence the message repeating over and over.

Is that still happening? In your logs irfan there is a big gap between the instances of the error message? The same sounds true in your case Jasmina - after the sharing violation the exact same file goes on the be successfully accessed many times.

If it does indeed recover itself when the layout reloads, then I think a quick retry loop would be sufficient to solve the issue. This is backed up by handle not showing any current “handles” in ifran’s test.

45

Yes, the client reports the error at the same moment sharing violation occurs, and no the error would actually NOT repeat in the client logs, it would show up once, and really only by looking at the display can you tell that something is wrong.

To summarize: the offending layout is indeed always the NEXT one in sequence and not the one on which the display stops, if there is a region on the current layout it will time out and disapear, leaving only the background image, if it is an image only layout, it will just show that indefinitely (until something is added to the schedule, priority slide comes up, or the client is restarted).

Not sure if it helps (or if it is indeed the universal case) but if you look at the PML log carefully, we noticed that the process sequence for the layouts is always create file/close file, with read and network query and other processes, thrown in there at the appropriate times, except at the instance sharing violation occurs.

So maybe the error itself is not the sharing violation but rather the lack of “close file” for XLF at that time (150 in this case). You can see the processes starting at 7:45:21 (or rather 6 by your time-zone) for XLF 150, the sequence is: create, query network…, close, create, query network…, close, CREATE, query standard…, read, CREATE; and that’s when the sharing violation occurs.

I cannot locate another place in the log where the same xlf is being created twice, without being closed in the meantime. As I said, not sure if it helps, just noticing :slight_smile:

46

OK, so the first thing to try and solve is why it gets stuck…

I’ve been working under the impression that something is locking the file and then keeping it locked each time Xibo tries to run the layout. However it seems more likely that it gets locked once as a one off and Xibo never tries to advance to the next layout as a result.

I’ll look into that first

47

OK, so I think I see why the file lock is causing the player to hang - I wont bore you with the details, but essentially it thinks it is already in an error condition, so it doesn’t need to raise another - which is incorrect.

I’ll get you an updated MSI in the PM shortly.

48

Thanks Dan, got the msi installed and will run it. Hoping for no issues. I will keep an eye on the log throughout the day and over the weekend to see if it gets stuck, but recovers.

Thanks Again for your work on such an amazing system

49

Thanks Dan, got the exe… installed and so far seems ok, on the player there were same errors, but as you mentioned it is moving to to next layout and getting refreshed, on the screens where there is only one layout assigned.

However we are not getting those errors registered on the CMS, and these gets reflected on the CMS only when the player is restarted, wish if the errors gets on the CMS once it happens…

keeping the watch over the weekend… and let you know the results…

50

Dear Friends,…

to start with… GOOD NEWS, after over 10 hrs of watch, if you look on the info window (Ctrl+I) on the player, it does show that the same errors have appeared… but it does not hang the player, and it does moves to the next layout, or refreshes the current layout (case of one layout only)…

Dan, you have hit the bulls eye… THANKS once again

hope this gets incorporated in the new releases as well…

so now i’ll be rolling out the new 1.7.8 EXE (sent on 28-05-2016), on all the players, with CMS 1.7.7.

:slight_smile: Cheers…

1 Like
51

They should be sent once the collection interval has expired, provided there are not more messages in the queue than can be sent in one go.

You should see it create log.xml files in the local library folder, which are essentially those error messages queued to send.

Excellent, that is great news! I am pleased we managed to get a sensible workaround in place, even if we can’t work out why the files are being locked!

I will wait for the others to comment before closing the issue -

52

Unfortunately, none of the players has sent any log files to the CMS, since yesterday, and sad part i could not see the log.xml in the C:\Xibo Library\

53

About 48 hours since we’ve updated all our offending clients to the custom release and absolutely no issues so far.

The problem is logged on some clients but otherwise no sign of anything out of the ordinary happening. The clients have also not logged the problem with the CMS but as long as they continue to run it is not really an issue for us. (We can investigate this further if you’d like.)

I would call this one solved; big thanks to Dan, everyone on this topic, and the whole Xibo team for doing such a great job on this project. Again, big thanks and keep up the good work.

1 Like
54

Hi, Can someone sent me the NEW client.exe to solve the above issue please.

55

I too would like this. I have a number of clients with this issue. Xibo appears to still be running but just a black screen. Keen to get new 1.7.8 in-place ASAP to hopefully resolve this issue.

56

I’ll add you both to the PM mentioned above.

I’m afraid that anyone else that wants it after that will have to wait a few days for me to build a proper release.

57

Yes Dan thank you so much! I too have gotten logged errors on certain players but they are still going strong with no more hangups on the stunned layouts.

Thanks Again!

1 Like