Download all raw data from Supla cloud

[thorstendb]

Hi Zamel,
is it possible to download the complete raw data from the cloud somehow?

[pzygmunt]

You can download the history of measurement data.

[thorstendb]

Yes, but there is (randomly?) missing data lines, can be found when comparing the timestamps.

[pzygmunt]

These are RAW data, i.e. the way we store them in the database. Client applications and Web UI fills these gaps according to: the algorithm below.

Records are made in the database every 10 minutes. If there is a larger distance, it means that the counter was offline. Let's assume that the break is 60 minutes and the difference between the points in the database is 600 kWh. Then we divide 600 / 6 and complete with 6 entries of 100 kWh each.

[thorstendb]

Hmmm ... not understood what should be the outcome.
In the data I downloaded as CSV or xlsx I saw, that entries were missing. Mostly 10 or 2x 10 minutes (so 1..2 entries on random locations in the list).
It seems that when using the "Counter" values, the overall values are correct. But when using the "Incremental" and adding everything up by myself, the values differ (some are missing).
I also found out, that the number of missing entries (complete lines) differ based on the PC and browser used.

I found them when trying to sum the things up by myself.
To find the gaps, added a column in excel and calculated each timestamp based on the first timestamp.

E.g.:
1707325971 2024-02-07 18:12:51
1707327171 2024-02-07 18:32:51

I wanted to build up some own data from the "Incremental" data, but everytime when a gap occur, the data gets wrong. And I am not sure about the Counter value.
I also compared the values to the "official" energy counter and the values seem correct when comparing to the Counter value.

[thorstendb]

I initially stumbled over it when adding values from my "official home counter" and the Supla to my comparison table, to see how good they match (they actually do very good).
I found out, that values I downloaded from supla cloud on one PC are different from values downloaded on another PC while filling my table.

Here is an example of the files I stored for that (can send if you like):
1706728371 2024-01-31 20:12:51 0,02908 0,0016 0,0109 0,01002 0,54838 0,0015 0,02038 0,1528 0,1598 0,08898 0,03732 0,08108 0,73726 0,09208 0,64518 0,73144 0,08626
1706730771 2024-01-31 20:52:51 0,02908 0,00162 0,0109 0,01022 0,61306 0,0015 0,02038 0,1713 0,16098 0,08898 0,03752 0,0811 0,80312 0,0921 0,71102 0,79728 0,08626

1706728371 2024-01-31 20:12:51 0,02908 0,0016 0,0109 0,01002 0,54838 0,0015 0,02038 0,1528 0,1598 0,08898 0,03732 0,08108 0,73726 0,09208 0,64518 0,73144 0,08626
1706728972 2024-01-31 20:22:52 0,02908 0,00164 0,0109 0,01022 0,63842 0,0015 0,02038 0,17194 0,17178 0,08898 0,0408 0,08108 0,83928 0,09212 0,74716 0,83342 0,08626
1706729571 2024-01-31 20:32:51 0,02908 0,00166 0,0109 0,01042 0,7141 0,0015 0,02038 0,19066 0,17354 0,08898 0,0413 0,08108 0,91672 0,09214 0,82458 0,91084 0,08626
1706730171 2024-01-31 20:42:51 0,02908 0,00168 0,0109 0,01062 0,78774 0,0015 0,02038 0,20964 0,175 0,08898 0,04152 0,0812 0,99182 0,09216 0,89966 0,98592 0,08626
1706730771 2024-01-31 20:52:51 0,02908 0,0017 0,0109 0,01082 0,85242 0,0015 0,02038 0,22814 0,17618 0,08898 0,04172 0,08122 1,05768 0,09218 0,9655 1,05176 0,08626

See that for "Forward active energy" we have:
PC 1: 2024-01-31 20:52:51 0,80312
PC 1: 2024-01-31 20:52:51 1,05768

That's why I tried the incremental (diff) values, but the results were even worse.

The results are also different when using Chrome or Edge. That's why I asked for the real raw data the webpage downloads to do it's calculations. How can I build up a complete database over all values?

[klew]

There was Supla Cloud and server update in recent days (version 24.02). There were some fixes included in this update, which fix few bugs in measurement history on Cloud.

Difference between browser my also be caused by those bugs. Measurements are downloaded from server to local database in browser, so if some data was broken in that local storage, then it could persisit and result in differences between different browsers.

After that recent update, browser should clear storage and download measurements again from server. So I hope that issues mentioned by you are now fixed. Please try again and check if problem is still visible.

[thorstendb]

Hi,
this sounds great.

I have cleared all my browser data.
Just downloaded and saw, that except for the first line the energy totals are not calculated - the columns are empty (browser: Chrome).
Same issue when using MS Edge. Download type: All, xlsx/csv, counter.

> Measurements are downloaded from server
Is it possible to fetch this data directly and do the calculations by my own?

When downloading values as Incremental, the missing values are calculated.
But when I sum them up, I get 259,26058 in excel and 259.51776 from webpage.
When summing up the time, there is one hour missing.

When summing up the three phases in "Counter" (where the sums are missing), the result looks ok compared to the webpage. But in the table all values except the first row do not have any decimals lower than 1:

1706721171 2024-01-31 18:12:51 0,00186 0,00016 0,0056 0,00002 0,01222 0,0013 0,00116 0,00374 0,00054 0,08066 0,00008 0,07338 0,01462 0,08212 -0,0675 0,0123 0,0798
1706721771 2024-01-31 18:22:51 616 36 650 180 1304 138 468 380 56 8312 8 7444 1716 8226

[fracz]

Just downloaded and saw, that except for the first line the energy totals are not calculated - the columns are empty (browser: Chrome).
Same issue when using MS Edge. Download type: All, xlsx/csv, counter

There was a mistake that resulted in these columns empty. In fact, the same error caused the lack of division of these values (no decimal points). It has just been fixed and will be released ASAP.

> Measurements are downloaded from server
Is it possible to fetch this data directly and do the calculations by my own?

You have to use the API for that. See https://svr1.supla.org/api-docs/docs.html and the getChannelMeasurementLogs endpoint.

[thorstendb]

Hi,

that sounds great! Please leave a note if this is ready for testing.
Did you also found the root cause why complete lines are missing?