1.7.7 player keeps crashing

Get Help
42

It would be very interesting to explore whether running 1.7.8 (or 1.7.7) with the webpage module set to Open Natively changes the OOM behaviour or not.

As far as I can tell, the same basic principles to open/close the webpage are used between 1.5.2 and 1.7.7 when 1.5.2 is set to offset 0, scale 100 and 1.7.7 is set to Open Natively.

Scaling and Offsetting does introduce slightly different behaviour because 1.7.7 made advances towards having the designer preview equal the actual output, which involved some scaling and transforming with JavaScript.

Ref: http://bazaar.launchpad.net/~dangarner/xibo/client-152/view/head:/client/dotNET/WebContent.cs#L84
Ref: https://github.com/xibosignage/xibo-dotnetclient/blob/develop/Media/IeWebMedia.cs#L75

43

For what itā€™s worth, I had to change to Manual Position because I needed to specify just the portion of the webpage that showed the hourly weather data, rather than the whole page.

I understand the test, and will work with that shortly. 1.5.2 is still running very nicely, in spite of changing the URL and altering the offsets slightly. I didnā€™t actually even shut the player down, but just made the changes and waited for the next change over, which is set to 900 seconds. I have another project I need to finish up, and will then implement the full page on 1.7.7 to see how it does with memory.

Edit: I just stepped over and opened Task Manager on the 1.5.2 player, and noted that, with the accuweather content, the memory has climbed nearly 1GB. It hasnā€™t crashed yet, but the localconditions page played nice for over 36 hours, as stated earlier, while accuweather does not appear to be playing nice. Apparently, not all javascript memory issues are created equal? More as I get itā€¦

44

Yeah sure, I appreciate that - there is absolutely a difference between the way the two players behave under those circumstances (offset/scale) whereas as far as I can see they are identical when it comes to opening a ā€œnativeā€ page. If that still leaks memory, then I am clearing missing something :slight_smile:

I think the .NET framework target changed between 1.5.2 and 1.7.7, which might cause different behaviour.

45

I also suspect that this might be the biggest part of the problem. Your use of the libraries will necessarily vary depending on the newer library entries.

Question: Is it possible to utilize legacy libraries in newer implementations of the .NET framework, or does that entail unnecessary risk? I have no experience with programming within the Microsoft world beyond a (very simple) school project several years ago, so I have no understanding of how they handle the libraries other than to say Iā€™m pretty sure they donā€™t remove older libraries in an effort to preserve backward compatibility. That may be incorrect, but it seems that I recall hearing or reading something along those lines once.

1.5.2 went OOM just a few minutes ago with the accuweather content. Time to run up 1.7.7 on full page and backtrack the 1.5.2 player to the localconditions page that did not OOM and start adding other content to it.

46

Ok, 1.5.2 is back on the localconditions page, scaled, and running very nicely again, using less than 1GB RAM consistently.

1.7.7 is running the accuweather page at full screen on Open Natively, and is currently using 1.22GB RAM. Will keep an eye on them both until tomorrow, when I will report back once again with the results of the latest tests.

47

Each time weā€™ve attempted to debug this issue before, its been like whack-a-mole and this time seems no different (aside from you persevering far longer than others :thumbsup:)

It sounds like full screen, open natively is still causing a problem, which takes all of the scaling and Xibo related manipulation out of the equation. We are effectively testing the WebBrowser control and its ability to open a webpage.

The next thing to try - if you still have the energy for it - is to increase the duration on the webpage item up to something very large (i.e. 3600 seconds - 1 hour).

This will prove whether it is the reload leaking memory or the page sitting at idle.

I donā€™t think it would be too risky, but its just not possible I am afraid. The WebBrowser control is essential an ActiveX control, which means we get what we are given with regards to the underlying dllā€™s. This also means it can vary from system to system.

48

There are two things driving me, here. First, I canā€™t stand leaving a computer problem unresolved. Computers are here to do what WE want, not what the computer wants. Second, my boss has me highly motivated to get this project completed, although he is not pressing for immediacy, which leaves me time to work with you, which satisfies the first itemā€¦

Unfortunately, I donā€™t yet know that open natively on 1.7.7 is actually causing the same problem, as the full page run got interrupted in the night because yours truly neglected to disable automatic Windows Updates.

Furthermore, a pop-up for moatad.js seems to have caused havoc with my last run on 1.5.2.

Is it BEER:30, yet?

I have them both running once again, with updates turned off (not sure what to do about z.moatads.com and that stupid pop-up, yet - will dig around the googles for that this morning), and will monitor memory usage on them as they progress.

I am very willing to alter the timing on my web content to 1 hour after I check the actual memory usage on the full page. I wonā€™t mess with it on 1.5.2, yet, as I am still working on adding content to the offset display in an effort to determine if it will fully meet my needs. However, I am more than willing to make that test happen on 1.5.2 afterward if you would like to compare the differences.

1 Like
49

We are certainly getting towards that point!!

Fab.

It sounds to me like you have a good idea of what constitutes a good test, so ill leave it in your capable hands. Thanks again.

50

OK, running the accuweather site on full page and open natively will still crash the player - eventually. When I started it yesterday morning, it started at just over 1GB memory and seemed to be staying pretty steady. However, as the day wore on, I noticed that it was very slowly climbing. When I left for the day, it had been playing for nearly 8 hours, and was up to 1.35 GB. I left it, and when I came in this morning, it was at 1.91GB, and has since climbed up to 1.94GB. This is not as alarming as the offset display, but it is definitely headed to a crash. I could easily build a reboot into Task Scheduler to resolve this, but it doesnā€™t display the rest of what I need, so except for providing data regarding behavior of the player, this test is moot.

Running the localconditions site on offset display on 1.5.2 continues to run at 1GB with no gain. I am still struggling with the stupid popup asking me to download moatad.js, but I believe thatā€™s not player related and nothing I can ask you to handle - unless you can utilize the IE popup blocker within the player. I would be surprised if you could, honestly. It turns out the popup did not cause any issues with the player running; the player was on time and chugging along just fine, so I simply closed the popup and let it continue.

I will go ahead and alter the timing on 1.7.7 from 15 minutes to 1 hour and monitor again. I suspect that the full page on open natively, with the longer duration, will not crash over the weekend, but that it will still experience growing memory usage. Time will tell.

I am also going to add the extra content to the 1.5.2 player and monitor the memory there. Not sure what to do about the popup, still, but I will do some digging into the google webs and see if I can find something. I expect the player to keep running regardless, but I canā€™t have that popup on the display if/when I run this out for production.

51

Just a quick note before the weekend; 1.7.7 has been running for nearly 7 hours with a 1 hour update on the full screen open native content. The memory has gone up from a starting point of 0.98GB to 1.04GB in that time. Thatā€™s significantly better as a starting point for comparison.

I will let these run over the weekend once again, and will see how they look on Monday.

52

Do you think we can assume that it is the refreshing of content causing the problem then? I.e. something is preventing the web browser control from freeing up memory when it is torn down.

This is by far the most ā€œpopularā€ bug reported with the WebBrowser control in .NET - what we can do about it, iā€™ve no idea.

It is really odd that you donā€™t have the same issue in 1.5 without and do in ā€œopen nativelyā€. I will trawl through the comparison again, but I am sure they are the same.

Edit: There is actually a small different it might be worth investigating.

1.5.2

1.7.8

53

First, my apologies for not posting results yesterday; I was caught up in too many other things to get information gathered and submitted.

Yes, and here is why:

1.7.7 - This is still up and running since Friday with the full page on open natively set to refresh hourly, but the memory has continued to be eaten up; it started at 0.98 GB and is currently at 2.08 GB. I am positive that this will eventually crash, but with the much slower rate of memory usage, I could easily reboot everything nightly and resolve the problem. I also found that the webpage I am displaying had crashed (not the player), and I had to refresh it to get it to continue displaying. That was odd, but I donā€™t think itā€™s related to the memory issue with player. The slower rate of refreshing the content seemed to dramatically slow the usage of memory, although the memory did definitely increase.

1.5.2 continued running for the same 4 days (Friday through Monday) without any significant rise in memory usage. I am still fighting with the stupid popup asking me to download moatad.js, but thatā€™s not your problem, really - itā€™s part of the stupid website Iā€™m using. Unfortunately, my preferred site causes even 1.5.2 to crash due to memory leaks, so Iā€™m still working on some kind of solution to this. I found it interesting to note that the web content is not being displayed as of this morning. It was over the course of the weekend (I was in the office for about 2 minutes on Saturday for something unrelated, so I checked in on this real quick), and it was present yesterday, but this morning the marquee that I have running is being displayed, but the web content is not. Memory started at 1.01 GB on Friday, and is now at 1.17 GB. If I can resolve the issue with moatad.js, I could be on my way to a working model of what I need to display, as I could also reboot this one on a nightly basis to keep both the memory usage down and the web content up if needed.

I just restarted the 1.5.2 player, and found that, in spite of the fact that the marquee was still scrolling across the page, the player was considered to have stopped working; Windows presented me with the ā€œwait for it to respond or close it nowā€ popup when I tried to close it out and restart it. After restarting the player, the web content is playing correctly. Again, this lasted from Friday until Monday, so I could easily reboot nightly and keep this one fresh once I figure out how to block Amazon Web Service from attempting to inject moatad.js into my system.

Interesting; I hadnā€™t yet gone in and checked the code for any of this, as I have so many other things I need to be doing. Please remember that, although I have a degree in software engineering, I have virtually no experience programming with .NET, and that it has been several years since I did more than BASH scripting on Linux servers. That being said, I took the time to check the code and do some external digging, which led to a question:

Is it possible to use .NETā€™s System.Runtime.InteropServices library? I found a page that talks about using that to clear the cache when an application hosts a WebBrowser control:

https://support.microsoft.com/en-us/kb/326201

I donā€™t know if that would help at all, and it is only the first of many results that I got in my searching, so I do not in any way expect that it will be the final resolution, but I thought it might be worth mentioning. I will continue to explore ways to possibly resolve this via code as I find some time here and there, though I donā€™t really believe that I will be able to contribute much. In the mean time, I am continuing my testing of both players, and may even backtrack a bit further through some older versions in an effort to get this to work.

Pre-post edit: I did some more digging on the whole Z.moatad.com/moatad.js download that keeps popping up. It turns out that the general consensus is to clear browser cache to get this to stop. Iā€™m not clear as to why, yet, but if thatā€™s the case, I wonder even more if the suggested library might help things out. I know I keep saying this, butā€¦

Iā€™m going to do some more digging.

54

There is an enormous amount of material on the web browser control and its penchant for leaking memory. Unfortunately the vast majority of it has proved to not work during the various development sprints in which weā€™ve tried to solve this issue.

Your investigation (which is much appreciated) has independently verified our previous conclusions - that each time we navigate away from a page it has the potential to leak memory. Whether it does or not depends on the JavaScript content of that page - which is why we can be 100% sure that Xiboā€™s own JavaScript for showing text/tickers/etc does not leak. (and why we can never be 100% sure that a 3rd party page wonā€™t leak).

The mystery between 1.5.2 and 1.7.8 is something hard to explain. The newer code improves the Dispose() method so that it only exposes managed resources when called manually, which I still believe is the right decision - it also drops a GC.Collect() because we found that to be ineffective. There are no other differences when opening natively, aside from the .NET framework version.

What is even stranger is that weā€™ve run your original layout for many days now and it does not leak on 3 different PCā€™s we have here as test players.

One of the suggestions weā€™ve not tried is to add this:

_webBrowser.Navigate("about:blank");

Mainly because it seemed stupid :smile: - however given we are on the edge here, iā€™ve put that in and built an EXE for you to try: https://dl.dropboxusercontent.com/u/353651/XiboClient.exe (iā€™ll remove this in a few days). Just replace your existing XiboClient.exe in your installation folder and restart the player.

The real solution to this issue to not to use the WebBrowser control - we definitely wonā€™t be using it in the new player development.

55

Thatā€™s very interesting; I found the following link yesterday, and dismissed it for pretty much the same reason you did - it seemed stupid:

However, as I have fewer ideas than you do, Iā€™m more than willing let you make that call. :grin:

That being said, I have put the sample player into place on the 1.7.7. machine and left 1.5.2 running as it was (Iā€™m still looking into that stupid popup on the older player - and may go bald over that oneā€¦). I will monitor this change through several hours or even into tomorrow for memory issues. At this point, Iā€™m leaving it at 1 hour and open natively on full screen for the first test. If I see different behavior, I will alter it to a shorter time and continue monitoring, then offset/scale it and monitor some more should the second test prove a change in memory consumption.

I want to state once again that I really appreciate the assistance with this. I feel itā€™s taking more of your time than I should be entitled to, and I am grateful for your willingness. I will continue to provide feedback for as long as you feel it will be productive.

56

If it does make a difference I will certainly feel a little stupid myself - although that wouldnā€™t make the solution any less stupid in my opinion :smile:

Thanks again for your assistance - I am not sure when it will be appropriate to call time on the investigation - I suspect we will both come to that conclusion at more or less the same time.

57

First, is this a case of K.I.S.S.? Why is it that the stupid/simple solutions are so often the correct ones, yet are most often passed over as invalid? I have done this myself in my own work, and yes, it always makes me feel stupid. (face-palm)

Second, while I in no way want to make you feel stupid, the stupid solution seems to be workingā€¦

The change has been running for at least three cycles, and no appreciable increase in memory. I noticed the memory spiked from 1.0 GB to 1.05 on the first cycle, but then it dropped back to 1.03 before cycling again, and has stayed there consistently ever since.

Assuming no appreciable change between now and tomorrow morning, I will change the frequency and let it run again. I apologize for the time it takes to test this all out; not only do I need to know how the player acts over a longer period of time, I also have quite a few other projects I have to be paying attention to through all this, which limits me a bit. This has has gotten very interesting, though!

58

If that little addition provides us with the answer it would be fantastic news (even it it only works on a subset of PCā€™s). I fear you might come back tomorrow and say it didnā€™t work!

Iā€™ve got my fingers crossed!

59

Sorry, but you can uncross your fingers. Hereā€™s what I see as I come into the office this morning:

The new player has gone from 1.03 GB up to 1.33 GB overnight. This isnā€™t huge, but is still an increase. Given the time it took, I would say thereā€™s really no difference. Iā€™m not sure why it seemed steady yesterday and then increased after I left, but itā€™s definitely the Xibo Client thatā€™s using the memory.

I also just discovered that I put the new player in on 1.7.8, so the watchdog is running. Iā€™m not sure thatā€™s relevant, but I needed to mention it.

I will go ahead and change the timing and continue monitoring this in the continued hope that it will be slow enough that a nightly reboot will resolve the memory problem.

60

I canā€™t say I am surprised - it does seem that there might be a marginal improvement though?

Yeah, it will run by default now - this isnā€™t anything to worry about and will be running in a separate process.

I think we are left with the two solutions mentioned before:

  • the new player development will not use the web browser control
  • the watchdog could be improved to try and detect high memory usage
61

Remember that I mentioned K.I.S.S., and that I mentioned how I tend to overlook the obvious. I kick myself for it, but I donā€™t learn very well. That being saidā€¦

I have continued to look into this as Iā€™ve had time, specifically looking for other ways to get the hourly weather data I have been requested to display. I finally found a different site that didnā€™t immediately crash the player, and started building from there. I ended up with a layout that includes two different videos that rotate, the date and time, a marquee text box, and the hourly weather I need. It didnā€™t crash, like I mentioned, but it did have an annoying popup that kept plaguing me - something about a certificate for some advertising site being invalid and would I like to continue or not. After some more digging which resulted in a great many places telling people to alter their registry and/or purchase some outrageously expensive software suite, I ended up coming up with a different solution; I altered the security/privacy settings for IE Internet Options. The result was significant, as the new layout plays for days on end without crashing or allowing popups in any way. I also was finally able to get Task Scheduler to successfully reboot the computer every night, and suddenly everything is working as I wanted it to all along.

From IE, go to Tools > Internet Options > Privacy > Settings and select Block all cookies (top position on the slider).

I also tried using High security level for internet zone on the Security tab, but that caused the hourly data to not be displayed at all (blank screen where it should display), so I set that back to Medium-high and everything started working the way I wanted.

Unfortunately, Iā€™m not sure if the security/privacy settings helped with the memory leak problem or not. I tried running the other URLs on the new settings, including the one my boss sent me as his preferred URL (http://www.intellicast.com/Local/Weather.aspx?location=USNY0011); the intellicast one crashes within an hour, while my preferred localconditions.com seemed to be much better, although it did slowly eat memory. It seems that the new URL, combined with the privacy setting to eliminate the popup, is the best bet, but Iā€™ve gone beyond my personal knowledge regarding what is actually causing the memory leak and why some are worse than others, such as openweathermap.org, which is what doesnā€™t appear to leak at all at this point.

At this point, I think Iā€™m willing to call this as close to resolved as Iā€™m likely to get until the new hourly section of the Forecast.io module is ready or the non-(web control) version is ready (or both!)

Once again, I truly appreciate all the help and investigation into possibilities. I look forward to playing with the new version once itā€™s ready.

1 Like