We’ve been successfully running Xibo in a multi-node Docker Swarm Mode cluster behind a load balancer for a few months now. Up to this point it’s only been myself and and the API connecting to the system, but just recently we began to allow more users access, and we’d like to try making the system more resilient when it comes to potential failures of the CMS.
Do you guys foresee any problems with running additional replicas of the CMS container? The entire filesystem is already mounted from NFS so they’d have the same files. My biggest concern would be session handling, but as long as that is done via cookies and the DB, I don’t think that would be a problem either.
Would love your input, thanks!
P.S. I’m not savvy enough with ZMQ to know for sure, but I assume that running replicas of XMR is off-limits. If I’m wrong, please correct me.
You should be OK running multiple replicas of the web container, providing as you say they are using shared storage for the library.
Sessions are handled in the database so there’s no issue there.
You may run in to issues with cache locking depending on how your NFS is setup. It’s not a configuration we run ourselves so I can’t comment how that will work in reality. You’ll have to try it and see.
For XMR to work, you’ll need all your web containers to be able to address its private port, and then all your Players to be able to see the same XMR instance’s public port.
Swarm should be able to enforce that one copy is always running somewhere, so I’d imagine that would be good enough?
XMR itself isn’t cluster aware as it’s stateless and you can just spin up another without loss.
Thank you for your help with this. I appreciate the time and thought you put into it, and I never responded.
As an update, it seems to be working fine. Swarm spreads the services across multiple containers based on resource availability and then spins up new ones as needed. I’ll report back if I discover any issues with the configuration, but the results are encouraging.
I would really be interested to know how he got Xibo to run in swarm mode. According to docker documents the project needs to be at version 3.0+ to do that.
The file version is really the only thing getting in your way, and thankfully they haven’t depreciated much in terms of config options so you can probably just switch to a newer version and use the compose file as is.
Thank you very much. I will run a test scenario and let you know how I come along. I am working on porting the storage for the library to a NFS share too.
The mount paths you see there are actually NFS and work like a charm. We chose to mount the NFS on the hosts rather than indivually mounting storage using the local nfs driver. It seemed to create unnecessary complexity to have each container start its own mount. Also, Swarm Mode mounts all volumes using bind mounts so if the directory cannot be mounted due to a failed host mount, it just moves to the next host.
I already have Xibo running on GCP but need High availability because we are going to have connections all over North America. Brodkin - I know you got it running with docker-swarm but is there any hope for Kubernetes? I can not find any documentation. The only thing I can come close to is Kompose.io converter, which by the way does not work. I could use any help I can get with swarm or Kubernetes.
Sorry, been away from this project for a sec… I don’t really have anything to add anyway since I’m not familiar with Kubernetes. That said, I don’t see any reason why it couldn’t work. It’s also possible to deploy a Docker stack directly to Kubernetes now, but I haven’t tried that either.