The command is not xibo-layer-test. It is xibo-player-test, and from the latest release xibo-player.
Newer versions also require you to specify whether you want them to connect over http or https.
Please go to the latest release blog post, and follow the instructions there. You must not be running as the root user, you must be a regular user for it to work.