XMR Failure : A resource was acquired at attached stack trace but never released

Fred_Ward · April 9, 2018, 9:20am

I am getting this message from all my displays regarding XMR. I am running Xibo Docker 1.8.7

E/StrictMode(22408): A resource was acquired at attached stack trace but never released. See java.io.Closeable for information on avoiding resource leaks.
E/StrictMode(22408): java.lang.Throwable: Explicit termination method 'response.body().close()' not called
E/StrictMode(22408):    at dalvik.system.CloseGuard.open(CloseGuard.java:184)
E/StrictMode(22408):    at java.lang.reflect.Method.invoke(Native Method)
E/StrictMode(22408):    at java.lang.reflect.Method.invoke(Method.java:372)
E/StrictMode(22408):    at okhttp3.internal.platform.AndroidPlatform$CloseGuard.createAndOpen(AndroidPlatform.java:340)
E/StrictMode(22408):    at okhttp3.internal.platform.AndroidPlatform.getStackTraceForCloseable(AndroidPlatform.java:155)
E/StrictMode(22408):    at okhttp3.RealCall.captureCallStackTrace(RealCall.java:89)
E/StrictMode(22408):    at okhttp3.RealCall.execute(RealCall.java:73)
E/StrictMode(22408):    at uk.org.xibo.player.c.onEventBackgroundThread(DisplayManager.java:1269)
E/StrictMode(22408):    at java.lang.reflect.Method.invoke(Native Method)
E/StrictMode(22408):    at java.lang.reflect.Method.invoke(Method.java:372)
E/StrictMode(22408):    at a.a.a.c.a(EventBus.java:498)
E/StrictMode(22408):    at a.a.a.c.a(EventBus.java:492)
E/StrictMode(22408):    at a.a.a.b.run(BackgroundPoster.java:64)
E/StrictMode(22408):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
E/StrictMode(22408):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
E/StrictMode(22408):    at java.lang.Thread.run(Thread.java:818)

alex · April 9, 2018, 10:40am

Thanks for your report.

Where are you seeing this error please?

dan · April 9, 2018, 10:44am

My thanks also!

I am curious how you’ve related that message to XMR?

Fred_Ward · April 9, 2018, 12:22pm

I saw this report on the Android Xibo client. I noted a XMR failure in the CMS log and was having issues finding the failure. So I submitted a rekey to a few of my clients and waited for the heartbeat. When it attempted to communicate with the server this was the result.

Fred_Ward · April 9, 2018, 5:33pm

I think I may have found the possible issue, or at least something that might be related to it.

ZMQ Exception: class org.b.f/Errno 4

Does this help at all?

Fred_Ward · April 9, 2018, 5:50pm

These are my config files:

indent preformatted text by 4 spaces
version: "2.1"
services:
cms-xmr:
    image: xibosignage/xibo-xmr:release-0.7
    ports:
        - "9505:9505"
    restart: always
    env_file: config.env
    mem_limit: 256m
cms-web:
    image: xibosignage/xibo-cms:release-1.8.7
    volumes:
        - "./shared/cms/custom:/var/www/cms/custom"
        - "./shared/backup:/var/www/backup"
        - "./shared/cms/web/theme/custom:/var/www/cms/web/theme/custom"
        - "./shared/cms/library:/var/www/cms/library"
        - "./shared/cms/web/userscripts:/var/www/cms/web/userscripts"
    restart: always
    links:
        - cms-xmr:50001
    environment:
        - XMR_HOST=cms-xmr
    ports:
        - "80:80"
    env_file: config.env
    mem_limit: 1g



indent preformatted text by 4 spaces
## CMS Configuration

## Please make a copy of this file as config.env, and then
## edit it to suit your environment

## MYSQL
# You're running with your own remote MySQL server
# Enter the connection details here that Xibo should use
# You'll need to ensure that the database already exists (and is empty)
# and the user specified has all privileges on that database
MYSQL_HOST=us-dc0xibodb01.domain.com
MYSQL_PORT=3306
MYSQL_DATABASE=xibo
MYSQL_USER=******
MYSQL_PASSWORD=********

## SMTP Server Configuration
## The CMS needs to be able to send email to you
## Please enter credentials for a suitable SMTP server
## Defaults will work for GMail - replacing your GMail username
## and password as appropriate. You will also need to enable access
## for less secure applications on your GMail account for this to
## work. See https://support.google.com/accounts/answer/6010255

## SMTP Server Hostname
#CMS_SMTP_SERVER=smtp.gmail.com:587
## SMTP Username
#CMS_SMTP_USERNAME=youraccount@gmail.com
## SMTP Password
#CMS_SMTP_PASSWORD=yourpassword
## Use a TLS Connection YES/NO
#CMS_SMTP_USE_TLS=YES
## Use a STARTTLS Connection YES/NO
#CMS_SMTP_USE_STARTTLS=YES
## Rewrite domain (the domain your email will appear to come from)
#CMS_SMTP_REWRITE_DOMAIN=gmail.com
## Hostname that we should identify ourself to the remote server as
#CMS_SMTP_HOSTNAME=gmail.com
## Can the From line be overridden in the outbound email
## NB GMail will rewrite the From address anyway so it's not important
## for GMail - YES/NO
#CMS_SMTP_FROM_LINE_OVERRIDE=YES

## It is sometimes necessary to configure the webserver running inside
## the container to know the DNS name by which you will normally
## access the CMS. For most installations this is unnecessary and can
## be left as default, however, if you know this, it won't hurt to
## set it
CMS_SERVER_NAME=xibocms.domain.com
CMS_ALIAS=/xibo

alex · April 10, 2018, 8:04am

What was the XMR failure you saw in the log? And what is it that isn’t working for you now.

Fred_Ward · April 10, 2018, 8:07am

All of the displays are having the xmr failure below:

onEventBackgroundThread - HeartBeatEvent XMR unresponsive, issue reconfigure

I have been able to resolve some issues while others remains… this is one of them. The other issue is that downloading did not want to work for the displays until I changed the download mode to off. However, downloading from the library takes time because it stalls. I know this is a different issue but I am working through it all a piece at a time.

More specifically, they would not download the javascript files and some user files.

alex · April 10, 2018, 8:12am

OK, so that means that the Players aren’t seeing heartbeat messages from XMR.

So either the XMR address they have been configured with is wrong, or XMR isn’t running, or there’s a firewall blocking access, or XMR itself is mis-configured.

Given it’s Docker, the only thing you need to configure is the XMR Public Address in the CMS settings. That should be

tcp://ip:9505

Where ip is the IP address of your server, or it can be the DNS name. You need to ensure that there is no firewall blocking access to that port.

Fred_Ward · April 10, 2018, 8:14am

Yep, it is. the dns host address, which is resolvable. I tried bringing down the firewall service in centos to test if firewalld was not parsing my services and ports correct but that killed docker networking.

alex · April 10, 2018, 8:16am

You can telnet to that port externally, and should be able to connect. Can you?

Fred_Ward · April 10, 2018, 8:20am

from localhost I can but not from and external host, such as my laptop

alex · April 10, 2018, 8:21am

Then something is blocking XMR - likely your firewall - and it can’t work until that is resolved. That is why the Players are unable to connect to the service.

Fred_Ward · April 10, 2018, 8:27am

That is what I think at this point too but damned if I know what rule is causing it.

indent preformatted text by 4 spaces
Chain INPUT (policy ACCEPT)

target prot opt source destination
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all – anywhere anywhere
INPUT_direct all – anywhere anywhere
INPUT_ZONES_SOURCE all – anywhere anywhere
INPUT_ZONES all – anywhere anywhere
DROP all – anywhere anywhere ctstate INVALID
REJECT all – anywhere anywhere reject-with icmp-host-prohibited

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all – anywhere anywhere
DOCKER-ISOLATION all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all – anywhere anywhere
FORWARD_direct all – anywhere anywhere
FORWARD_IN_ZONES_SOURCE all – anywhere anywhere
FORWARD_IN_ZONES all – anywhere anywhere
FORWARD_OUT_ZONES_SOURCE all – anywhere anywhere
FORWARD_OUT_ZONES all – anywhere anywhere
DROP all – anywhere anywhere ctstate INVALID
REJECT all – anywhere anywhere reject-with icmp-host-prohibited
DROP all – anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
OUTPUT_direct all – anywhere anywhere

Chain DOCKER (3 references)
target prot opt source destination
ACCEPT tcp – anywhere 172.19.0.2 tcp dpt:9505
ACCEPT tcp – anywhere 172.19.0.3 tcp dpt:http

Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
DROP all – anywhere anywhere
RETURN all – anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all – anywhere anywhere

Chain FORWARD_IN_ZONES (1 references)
target prot opt source destination
FWDI_public all – anywhere anywhere [goto]
FWDI_public all – anywhere anywhere [goto]

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target prot opt source destination
FWDI_public all – 10.12.76.0/24 anywhere [goto]
FWDI_public all – 10.12.72.0/24 anywhere [goto]

Chain FORWARD_OUT_ZONES (1 references)
target prot opt source destination
FWDO_public all – anywhere anywhere [goto]
FWDO_public all – anywhere anywhere [goto]

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target prot opt source destination
FWDO_public all – anywhere 10.12.76.0/24 [goto]
FWDO_public all – anywhere 10.12.72.0/24 [goto]

Chain FORWARD_direct (1 references)
target prot opt source destination

Chain FWDI_public (4 references)
target prot opt source destination
FWDI_public_log all – anywhere anywhere
FWDI_public_deny all – anywhere anywhere
FWDI_public_allow all – anywhere anywhere
ACCEPT icmp – anywhere anywhere

Chain FWDI_public_allow (1 references)
target prot opt source destination

Chain FWDI_public_deny (1 references)
target prot opt source destination

Chain FWDI_public_log (1 references)
target prot opt source destination

Chain FWDO_public (4 references)
target prot opt source destination
FWDO_public_log all – anywhere anywhere
FWDO_public_deny all – anywhere anywhere
FWDO_public_allow all – anywhere anywhere

Chain FWDO_public_allow (1 references)
target prot opt source destination

Chain FWDO_public_deny (1 references)
target prot opt source destination

Chain FWDO_public_log (1 references)
target prot opt source destination

Chain INPUT_ZONES (1 references)
target prot opt source destination
IN_public all – anywhere anywhere [goto]
IN_public all – anywhere anywhere [goto]

Chain INPUT_ZONES_SOURCE (1 references)
target prot opt source destination
IN_public all – 10.12.76.0/24 anywhere [goto]
IN_public all – 10.12.72.0/24 anywhere [goto]

Chain INPUT_direct (1 references)
target prot opt source destination

Chain IN_public (4 references)
target prot opt source destination
IN_public_log all – anywhere anywhere
IN_public_deny all – anywhere anywhere
IN_public_allow all – anywhere anywhere
ACCEPT icmp – anywhere anywhere

Chain IN_public_allow (1 references)
target prot opt source destination
ACCEPT tcp – anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:sunrpc ctstate NEW
ACCEPT udp – anywhere anywhere udp dpt:sunrpc ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:nfs ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:snmp ctstate NEW
ACCEPT udp – anywhere anywhere udp dpt:snmp ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:2377 ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:7946 ctstate NEW
ACCEPT udp – anywhere anywhere udp dpt:7946 ctstate NEW
ACCEPT udp – anywhere anywhere udp dpt:4789 ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:http ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:9505 ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:cslistener ctstate NEW
ACCEPT tcp – anywhere anywhere tcp dpt:50001 ctstate NEW

Chain IN_public_deny (1 references)
target prot opt source destination

Chain IN_public_log (1 references)
target prot opt source destination

Chain OUTPUT_direct (1 references)
target prot opt source destination

Fred_Ward · April 10, 2018, 8:28am

It does not appear to be an external firewall either because I ssh to the db server on that same subnet no problem.

alex · April 10, 2018, 8:29am

You don’t want to allow the outside world to access port 50001. Only the CMS should have access to that, and indeed Docker doesn’t expose it, so I don’t think that rule will actually do anything, but just as a point of note.

I’m afraid it’s going to be a case of going through the firewall logs and seeing you can track down what is blocking access. It’s definitely just tcp port 9505 that is required.

Fred_Ward · April 10, 2018, 8:31am

Yeah, I was less concerning at the moment while trying to work though this. That was part of a desperate attempt to figure all this out. It is gone. Did no good anyway. The source 10.12.76.0/24 was part of that attempt too… I thought that the firewall was blocking my displays so I opened up one of the ranges for testing.

Fred_Ward · April 10, 2018, 8:42am

Last but certainly not least… ring a network guys neck and shake him like a dead chicken. Maybe that will resolve the issue, lol.

Fred_Ward · April 10, 2018, 1:48pm

I just talked the network team and this server resides outside the firewall.

Blockquote
The server is outside the firewall and firewall rules would have no effect of the traffic going to this server. The ports that should be open on the Xibo server are TCP/80,TCP/443, and TCP/9505; the only port I cannot connect to is TCP/9505.

alex · April 10, 2018, 1:49pm

Unfortunately it doesn’t change the fact that you can’t connect to the port, so something there is blocking it.

That you can connect from the server itself, but not externally implies to me that there’s a firewall or something blocking access.