Can't upload large files

xibologic · July 20, 2020, 7:54am

Hi,

I recently had to upload a lot of large files via the API, and found that files greater than ~2.5 gigs can’t be uploaded.

However, it works fine within Xibo, and all my php.ini settings and config.env settings are good.

Here is the error :

> **Notice** : Undefined offset: 1 in **/var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php** on line **104**
> 
> **Notice** : Undefined offset: 1 in **/var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php** on line **105**
> :xmedia printed :
> **Notice** : Undefined variable: xMedia in **/var/www/html/wioo-xibo/.public_html/vendor/xibosignage/oauth2-xibo-cms/objects/media.php** on line **63**
> 
> **Fatal error** : Uncaught InvalidArgumentException: Status code must be an integer value between 1xx and 5xx. in /var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/psr7/src/Response.php:151 
Stack trace: 
#0 /var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/psr7/src/Response.php(98): GuzzleHttp\Psr7\Response->assertStatusCodeRange(0)
#1 /var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php(116): GuzzleHttp\Psr7\Response->__construct(0, Array, Object(GuzzleHttp\Psr7\Stream), NULL, NULL) 
#2 /var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php(49): GuzzleHttp\Handler\StreamHandler->createResponse(Object(GuzzleHttp\Psr7\Request), Array, Object(GuzzleHttp\Psr7\Stream), NULL) 
#3 /var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(66): GuzzleHttp\Handler\StreamHandler->__invoke(Object(GuzzleHttp\Psr7\Request), Array) 
#4 /var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/guzzle/src/Middleware.php(30): Guzzle in **/var/www/html/wioo-xibo/.public_html/vendor/guzzlehttp/psr7/src/Response.php** on line **151**

PHP can clearly see the file, as it still is uploaded to the CMS, however it stops the code and I can’t get arround it…

dan · July 20, 2020, 8:11am

Hi,

The file mentioned here isn’t part of Xibo as far as I am aware, there isn’t even any /objects/ folder in that library.

Also I think StreamHandler is used by Guzzle when curl isn’t available, which is strange as curl is included in our containers.

The error itself implies that the response didn’t contain any headers at all, as for why that might be, I am not sure. Are there any Apache errors that match up with the above one?

All in all something looks a bit strange with the environment?

Thanks,
Dan

xibologic · July 20, 2020, 8:21am

Thanks for the quick response.

The errors related to the xMedia being undefined are “normal”

I created the folder “objects” with the media.php file for tests purposes, inside of it there is a “create” function, that call the XiboLibrary->create() function and stores the result in the $xMedia file.

As you can see on the image above, I’ve been trying to work arround the error. So the undefined xMedia variable comes from the “finally” statement. However I don’t think that it has anything to do with the problem, considering the errors still happens when I remove the “finally” part

I’ve looked in the apache error logs, and it’s just a copy of what was written on the web page, with some more logs :

[Mon Jul 20 10:08:39.270539 2020] [mpm_prefork:notice] [pid 648] AH00169: caught SIGTERM, shutting down
[Mon Jul 20 10:08:40.118713 2020] [mpm_prefork:notice] [pid 31740] AH00163: Apache/2.4.38 (Debian) configured – resuming nor
mal operations
[Mon Jul 20 10:08:40.118779 2020] [core:notice] [pid 31740] AH00094: Command line: ‘/usr/sbin/apache2’
[2020-07-20 10:08:43] API LIBRARY.INFO: Getting list of Displays
[2020-07-20 10:08:43] API LIBRARY.DEBUG: Passing GET params to request
[2020-07-20 10:08:43] API LIBRARY.DEBUG: Creating a new request with received parameters
[2020-07-20 10:08:43] API LIBRARY.DEBUG: Getting Authenticated Request
[2020-07-20 10:08:43] API LIBRARY.DEBUG: Checking Access token and requesting a new one if necessary
[2020-07-20 10:08:43] API LIBRARY.INFO: Getting a new Access Token
[2020-07-20 10:08:43] API LIBRARY.DEBUG: Getting parsed response from Abstract Provider
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Hydrating the response
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Preparing media payload
[2020-07-20 10:08:44] API LIBRARY.INFO: Uploading new media file 5f15510c2db207.73324521
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Passing POST params to request
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Creating a new request with received parameters
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Getting Authenticated Request
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Checking Access token and requesting a new one if necessary
[2020-07-20 10:08:44] API LIBRARY.INFO: Getting a new Access Token
[2020-07-20 10:08:44] API LIBRARY.DEBUG: Getting parsed response from Abstract Provider

Regarding this :

API LIBRARY.INFO: Uploading new media file 5f15510c2db207.73324521

The numbers comes from the uniqid() PHP built-in function, so there is no duplicated names.

dan · July 20, 2020, 8:36am

OK that makes sense then.

This means an Apache process has been killed, but appears to be from a previous log message. In fact all of the stuff above

[2020-07-20 10:08:44] API LIBRARY.DEBUG: Preparing media payload [] []

Must come from other parts of your code which you aren’t showing?

This is a hunch only, but if you’re running this script on the same server, processing a big loop of files and making multiple API requests, you might have a memory leak in your script which eventually consumes all the resources on the box.

Can you rule that out by quickly spinning up a php:cli container with your import?

xibologic · July 20, 2020, 8:38am

I’m sorry I don’t really understand what you want me to do ?

dan · July 20, 2020, 8:47am

You’re running the import script and the CMS on the same server, correct?

If yes, then i’d like you to separate those two things so that you run the Import Script on a different server to the Xibo CMS. I think the two things are interacting with each other, namely that there is a memory leak importing multiple files in a loop.

A simple example would be a new folder on your machine called /import, containing your script which does the import.

Open a cli inside that folder and then run:

docker run --interactive --tty --volume $PWD:/app --volume ~/.composer:/tmp composer require xibosignage/oauth2-xibo-cms

docker run -it --rm --name my-bulk-import -v "$PWD":/usr/src/myapp -w /usr/src/myapp php:7.4-cli php import.php

What you’ve got then is a completely separate PHP environment with runs your script. You will need to configure your import to use the Xibo CMS’s public URL.

This is just a hunch, but I can’t think of anything else it might be

xibologic · July 20, 2020, 9:11am

No luck with that either, still get the same error messages…

Do you have any way to try uploading a file greater than 2.5 gigs to a CMS via API ?

I am wondering if the problem occures only on my end.

thor57 · August 5, 2020, 1:57pm

I am having the same problem. This is my post from April, 2.3.3 upload errors. As you can see from my post, I only started encountering the error after upgrading my server. I haven’t found a solution yet. I am trying to find the time to install an older version on a test machine to try to troubleshoot further.

xibologic · August 10, 2020, 8:04am

Hi,

Did you had any chance to try it on your end ?

Have a nice day.

dan · August 10, 2020, 8:59am

I haven’t tried it yet, no… I will do my best to make time and get it tested out.

I think the only things which will have changed between versions are the base image for the container and the php/apache versions. None of the settings or logic on the CMS side itself have changed.

dan · August 10, 2020, 2:18pm

I have just managed to give this a try - with a 2.7GB file and it was uploaded no problem. I did have to change config.env to set the CMS_PHP_POST_MAX_SIZE and CMS_PHP_UPLOAD_MAX_FILESIZE to 4G, but other than that it worked without issue.

Can you try from the UI to rule out anything API related?

xibologic · August 11, 2020, 4:46pm

Hi,

I had already change the CMS_PHP_POST_MAX_SIZE and CMS_PHP_UPLOAD_MAX_FILESIZE to a greater value, aswell as some other lines in php.ini

From the UI it works great. In my opinion, the problem doesn’t come from there, as the file is still uploaded, the only problem is that an error is thrown and that stops the process of my script (even with try/catch)

dan · August 21, 2020, 8:08am

The user interface calls the same API as you are calling I think the issue maybe your script then?

Can you upload it to a public repo somewhere so that we can see it?