Remote Datasets Do Not Truncate

Hello gentlemen,

We’re running two remote datasets, but the data in one is getting duplicated even though Truncate is set to always. On our other dataset we have similar but different data. It no longer will update the data at all in spite of saying 27 rows were found during the test.

Both remote datasets use the same source URL so is there some sort of URL-based cache that might be commingling the data?

CMS Version: 1.8.9

Update: This problem was happening on 1.8.7 initially and I updated to 1.8.9 to see if that would resolve anything. I truncated the data in each table and then when the task ran both datasets started showing that they were being updated, but now both tables are completely empty. Once again, the test script finds all of the rows as expected:

"messages": [
      "Processing 1 results into 6 potential columns",
      "Processing Result with Data Root ",
      "DataRoot is an array",
      "Processing 0",
      "Processing 1",
      "Processing 2",
      "Processing 3",
      "Processing 4",
      "Processing 5",
      "Processing 6",
      "Processing 7",
      "Processing 8",
      "Processing 9",
      "Consolidating entries",
      "There are 10 entries in total"

I had the same issue, I fix the problem after deleting cache folder, inside $path/cms/library/cache/*, I don’t know if it is only coincidence, but after that, dataset works.

Thank you

Okay, so it suddenly started updating on its own like an hour ago BUT it’s importing the entire dataset twice for each of them. @Kleber, did you have that problem as well?

Yes, I had that problem in xibo 1.8.7, after upgrade to1.8.9, the issue gone. But I had this issue only when I had any changes in the value of json response, xibo duplicated the data, one with oldest values and one with new values.

It may highly depend on the truncate and refresh rates that you’ve set on your dataSet.

While it is true that in previous CMS version (1.8.7) the truncate didn’t work and we’ve fixed that, there is still an issue with the logic and calculations there, especially if you have say refresh daily and truncate every week or similar -

Interesting. I was running 1.8.7 and then upgraded to 1.8.9 but the behavior was the same. That’s when I opened this thread. Ultimately, clearing the cache caused the DataSets to begin acting more normally. I’m guessing that the buggy code was stored in the cache.

@Peter, what is the purpose of the cache and is there any reason why it is persisted to disk? It seems like a tmpfs would be more appropriate since you’d always want to start clean based on the latest code.

It is generally always good to clear the cache after upgrade (in most cases ctrl+F5 in browser should suffice, especially for the frontend changes), although as you said, some things might be cached in the cache folder in CMS.

This particular change to fix the truncate in 1.8.8, was a very small change in the dataSet entity, I don’t think I’d expect it to stay cached after upgrade - although what you said would imply that it did or perhaps something else with the task itself was cached.

Perhaps there could be some improvements made to the way cache is cleared after upgrade, generally cache is more complex that one may think. Certainly something to think about though.