First sensor data loads

[jaketbouma]

First data loads and investigation

As a data engineer I would like to investigate and consolidate the initial raw csv files provided on google drive so that it is easier for others to get started with exploration and analysis of the data

Definition of done

• Generate an inventory of files
• Document work and make a "getting started" notebook / sandbox directory
• Generate summary statistics (number of measurements, sample rate)
• [ ] Chase down metadata for available fields (There is no metadata, but the fields are mostly well formed)

Next steps

• Data quality (check for gaps in sensors)
• Data exploration
• Baseline modelling

[jaketbouma]

Wrote out multiple formats. The notebook 20180928 - Merge local data.ipynb syncs down the data and merges all the different formats.
Some things may still have crept through, we'll hopefully see them as we go.

To get started analyzing the data, you can pull up 20181002 - Getting started with prepared datasets.ipynb

[jaketbouma]

Summary statistics

from 20181002 - Getting started with prepared datasets.ipynb

Column fill

Number of nulls per column

Air temp           0
Humidity           0
Water level        0
Water temp         0
EC                 0
pH                 1
CO2             8589
DO             37185

number of records

dead times

conclusion

Data looks more than good enough to get going

[Patechoc]

There is actually many more months of measurements than I thought ;)

[jaketbouma]

☝️ See above corrected plots