Hello, we have rebuilt our Xibo instance twice now and both times the weather widget works for a couple days and then breaks. Basically, the weather widget starts showing “Unexpected Error, please contact support.” in the layout previews. And shortly thereafter, the clients that show weather data start to show nothing but a black screen. Logs start to show “[47.html] Download error: [HTTP] 500 Internal Service Error”. API calls to dark sky completely stop.
Natasha, from what I can tell, we have everything setup correctly. I am using the templates already provided within the Widget. Additionally, we have the API setup correctly in our DarkSky and Xibo portal. I have even added the “Powered by DarkSky” into the layout name like the disclaimer says. Still a no go.
When we first set everything up, the widget worked perfectly and as expected for 3 or 4 days. Then it just quits with the errors I provided.
Side note: Logging into the DarkSky portal today I see that they have been purchased by Apple. But will be providing API subscribers with access until the end of 2021.
Yeah we also saw that this morning, its bad news for Xibo users, but we will find an alternative source of data.
I don’t think your problem is related though, if you already have an API key you are good to use that until 2021.
[47.html] Download error: [HTTP] 500 Internal Service Error
This should show up in the CMS logs under the “XMDS” channel filter - please can you have a look in there to see if there are any errors reported. Or alternatively use the report fault process to get a troubleshoot.zip file.
Dan, here is what I am seeing in the logs under XMDS -
SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction
I have very few displays, 6 right now to be exact. None of them should have the same hardware key and none of them are showing the same layouts (2 ppts, 2 web, 1 weather, 1 testing). There is only one showing the weather layout.
I will verify that none are sharing a hardware key.
Cache is set to 60/m
I actually haven’t gotten any deadlock errors today. I had the weather layer turned off until about 20 min ago when I started testing for you again.
Here is what I see in the troubleshooting logs when I try to go into the layer and do a “Get Forecast” call and then re-publish the layout -
Unknown error during getResource. E = Concurrent record locked, time out.
[55.html] Download error: [HTTP] 500 Internal Service Error
Unable to reach Forecast API: cURL error 28: Operation timed out after 20001 milliseconds with 0 out of 0 bytes received (see libcurl - Error Codes)
Unable to reach Forecast API: cURL error 28: Operation timed out after 20001 milliseconds with 0 out of 0 bytes received (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)
That’s your problem I suspect - until you get one successful call you’ll get locks (so that we don’t repeatedly call the API).
Is there any reason your CMS wouldn’t be able to get access to the API? Can you, for example, make a request from the host machine successfully?
Yes. We have had many successful calls to the API from the Xibo Ubuntu server itself and from within the container. As I mentioned in the first post, everything works great as a fresh new server. But let it sit for a couple days, and it breaks. We rebuilt it twice to prove this out.
When it happens again, can you try restarting the container rather than rebuilding it?
It seems to me that a request timeout after 20 seconds would indicate a network issue from inside the container - Xibo uses curl to call the DarkSky API.
This has all been rewritten in 2.3.3 (due to DarkSky going away), so it may well be worth trying an upgrade.
Honestly, I don’t think updating will help, but its worth a try anyway.
Everything i’m reading about cURL error 28 points to firewall, dns or other network problems. It sounds like its a networking problem in the container, but why that wouldn’t be solved on restart I don’t know.
You might need to exec into the container and run a nslookup/ping command to the API to see if its still accessible or not.
