For WIndows:
We’ve successfully run an RTMP (via Open Broadcaster Software to an niginx server that accepts RTMP) feed that goes through a video.js player embedded in a simple HTML file on the nginx server. Because we only have a few displays, I configure the Xibo box to pull double duty (e.g. running CMS and streaming). We then embedded HTML in a Xibo content region & iframe’d the video player page in. Took a little tweaking on content region size, etc. but so far everything works like a charm.
Not sure if you have the option to change the feed type on your streaming server. If so, the above strategy may work for you, too. If not, there are other Javascript packages that will play RTSP streams (flowplayer, possibly?) and you can then still use the iframe method.