Skip to content

04. Implementing Rain Gauge Statistics

Jump to bottom
Matthias Prinke edited this page Jun 20, 2024 · 11 revisions

Initial Concept

Currently, the rain gauge value is provided as-is, 0...999.9 mm for the "5-in-1" protocol, 0...19999.9 for the "6-in-1" protocol. When an overflow occurred in the former, my main and only customer (Dad!) thought that he had found a bug in my code! (Whaaat? 😮) After some explanation, he stated that it would be nice to have the same kind of data on the smartphone's MQTT Panel as on the Bresser weather station display. Here comes the feature request...

The following problems have to be addressed:

  • Which values are wanted/needed?

    • Rate - Current rainfall rate (base on 10 min rain data) [not implemented]

    • Hourly - the total rainfall in the past hour

    • Daily - the total rainfall from midnight (default)

    • Weekly - the total rainfall of the current week

    • Monthly - the total rainfall of the current calendar month

    • Total - the total rainfall since the last reset [not implemented]

  • Which range can be expected from those values?

    see Wikipedia's list of weather records - Section "Rain"

  • Which algorithms are needed? Which data has to be accumulated in nonvolatile memory? In any case, an overflow of the rain gauge value has to be taken into account. The maximum rain gauge value has to be provided and the number of overflows must be stored.

    • Daily/weekly/monthly rainfall Pseudocode:
    foreach x in {day, week, month} {
    if (tsPrev.<x> != tsNow.<x>) {
        // day/week/month has changed
        // save timestamp at begin of new interval 
        tsBegin_<x> = tsNow;
        // save rain gauge value at begin of new interval
        rainBegin_<x> = rainNow;
    }
    rain_<x> = rainNow - rainBegin_<x>;
}

    Note:

    The begin of an accumulation interval (daily/weekly/monthly) is not detected exactly if the algorithm is executed at an interval (e.g. 10 minutes). In this case, the maximum deviation of the actual (timestamp;value)-pair from the ideal (timestamp;value)-pair is the length of the execution interval and the maximum possible change of value in this interval, respectively.

    • Total rainfall in the past hour

  • How does this data fit into the limited LoRaWAN payload?

    Currently all four values are transmitted as float (i.e. 4 bytes each). If required, 16-bit fixed point values could be used, but care has to be taken that no overflow occurs (considering global extremes).

  • How to get the date and time of day (e.g. from the LoRaWAN network)?

  • Which re-syncing interval of the RTC to the network time is reasonable?)

    Assuming 24 hours, can be configured with CLOCK_SYNC_INTERVAL in BresserWeatherSensorTTN.cfg.

  • Do we have to consider daylight saving time?

    Yes, if we want to calculate daily (and monthly) rainfall correctly.

  • Implement reset of rain gauge statistics by LoRaWAN downlink message ✔️

New Hourly Rainfall Algorithm

Old Hourly Rainfall Algorithm

  To determine the rainfall in the past hour, timestamps and rain gauge values are stored in a circular buffer:

       ---------------     -----------
  .-> |   |   |   |   |...|   |   |   |--. 
  |    ---------------     -----------   |
  |     ^                   ^            |
  |    tail                head          |
  `--------------------------------------'
  • new value: increment(head); rain[head] = rainNow; ts[head] = tsNow;
  • remove stale entries: if ((ts[head]-ts[tail]) > 1hour) { increment(tail); }
  • calculate hourly rate: rain_hour = rain[head] - rain[tail];

ts: timestamp

The required size of the circular buffer is determined by dividing one hour by the minimum sensor data reception interval, e.g. for 1 hour at an interval of 6 minutes, 10 entries are needed. For https://github.com/matthias-bs/BresserWeatherSensorTTN, only the sleep interval is fixed, the sensor reception interval and the LoRaWAN transmission interval are variable (but limited):

t_sleep t_rx_sensor ... t_LoRaWAN ...
SLEEP_INTERVAL < WEATHERSENSOR_TIMEOUT < SLEEP_TIMEOUT_JOINED (min.)
or
< SLEEP_TIMEOUT_INITIAL + SLEEP_TIMEOUT_JOINED + SLEEP_TIMEOUT_EXTRA (max.)
  • Result
Value NV data
Hourly ts[BUF_SIZE], rain[BUF_SIZE], head, tail
Daily tsBegin_day, rainBegin_day
Weekly tsBegin_week, rainBegin_week
Monthly tsBegin_month, rainBegin_month
ts_prev

In the actual implementation, the following optimizations have been made to save non-volatile memory:

  • Hourly: timestamps are stored as seconds since midnight, rain gauge values are stored as 16-bit fixed point data (1 decimal)
  • Daily/Weekly/Monthly: only the required parts of a timestamp (i.e. day, week or month) are stored

Implementation using ESP32 RTC RAM

The initial implementation uses the ESP32 RTC RAM to store data during deep sleep mode.

Pros:

  • No wearing of memory cells
  • Fast access

Cons:

  • Currently not implemented on ESP8266 (due to different API)
  • Data is lost after power-off or reset

Implementation using NVS (Non Volatile Storage)

Using non-volatile storage on the ESP32/ESP8266 utilizes the flash memory, which has a limited number of write cycles.

See Flash memory durability (SPIFFS), NVS Read/Write Over Usage and SPIFFS write cycles limit? for a primer.

The Arduino ESP32 Preferences library (also see arduino-esp32/tree/master/libraries/Preferences) is a wrapper around the Non-Volatile Storage library. The Preferences library by Volodymyr Shymanskyy is a compatible implementation for ESP8266, RP2040 and other platforms.

See Non-Volatile Storage (NVS) FAQ. NVS performs a write access only if the item to be updated has actually been modified; see nvs_storage.cpp.

Clone this wiki locally