DataSets: Truncate even if no data is returned

KluthR · August 18, 2021, 5:02am

xibosignage/xibo-cms/blob/cec885a4ab214198bd5dae6183b3ea041b8a743b/lib/XTR/RemoteDataSetFetchTask.php#L117

      
        
            $dependant = null;
            if ($dataSet->runsAfter != null && $dataSet->runsAfter != $dataSet->dataSetId) {
                $dependant = $this->dataSetFactory->getById($dataSet->runsAfter);
            }
            
            
$this->log->debug('Fetch and process ' . $dataSet->dataSet);
            
            
$results = $this->dataSetFactory->callRemoteService($dataSet, $dependant);
            
            
if ($results->number > 0) {
                // Truncate only if we also fetch new Data
                if ($dataSet->isTruncateEnabled() && $runTime >= $dataSet->getNextClearTime()) {
                    $this->log->debug('Truncate ' . $dataSet->dataSet);
                    $dataSet->deleteData();
            
            
        // Update the last clear time.
                    $dataSet->saveLastClear($runTime);
                }
            
            
    $rowsToAdd = $results->number;
                $this->log->debug('Current number of rows in DataSet ID ' . $dataSet->dataSetId . ' is: ' . $currentNumberOfRows . ' number of records to add ' . $rowsToAdd);

Why is that? I wondered, why I always have entries but the data source offers none. Now I found, that Truncation only executes when at least one NEW dataset is returned.

Would highly welcome a change here to say, that Xibo should use the datasource “as-is” - even if no data is available currently.

KluthR · August 31, 2021, 4:24am

Anyone of the staff could check this? I have to manually delete the data every day (if there is no data for the day) in order to get the old out of sight. This behaviour was definitely not there before the Upgrade to 3.0.2 (upgrading from 2). There must be something going on

dataset

DataSetID	DataSet	Description	UserID	lastDataEdit	code	isLookup	isRemote	method	uri	postData	authentication	username	password	customHeaders	userAgent	refreshRate	clearRate	runsAfter	dataRoot	lastSync	summarize	summarizeField	lastClear	ignoreFirstRow	sourceId	rowLimit	limitPolicy	folderId	permissionsFolderId
3	Raumverwaltung PROD		1	1630383626		0	1	GET	http://raumverwaltung.jki.intern/XiboOutputV2.php		none					0	1	0		1630380602	none		0	0	1	\N	stop	1	1

KluthR · September 23, 2021, 4:53am

@dan @alex Any advice from devs side? I dont understand how the last update managed to break this behaviour, since the code above was not changed at that point for years

natasha · September 24, 2021, 9:54am

Hi, thank you for posting.

I have taken this to the development team to take a look at. As a result they have created the following issue:

github.com/xibosignage/xibo

Remote DataSets : add new option to truncate dataSet data even with no new data returned

opened 09:00AM - 24 Sep 21 UTC

PeterMis

enhancement

In version 2 due to incorrect workflow, regardless of the number of returned ent…ries from the remote dataSet the data was truncated (if truncate was enabled). In version 3 we've fixed the workflow issue, as such the data will only be truncated when new data is returned (and truncate is enabled). However, even though the workflow is correct in version 3, some users may not expect this change, as such it might be desirable to implement a new option to remote DataSet configuration that would allow for it to be truncated regardless if new data is returned or not.

Thank you for bringing this to our attention, much appreciated