NOAA ISD data documentation

[lauramazzaro]

I am trying to use the NOAA Integrated Surface Data from the Data Catalog, and I haven't been able to find the documentation necessary to work with it. There aren't any example notebooks, and there isn't any information about how the data is stored within the blob container provided.

I was able to explore the file storage structure within the storage blob using the Storage Explorer, and I am still not clear on where to find the files that I need.

The specific file that I have been looking for in this example corresponds to station number 72365623049.

These are the files I've reached so far, that looked promising:

1. https://noaaisd.blob.core.windows.net/noaa-isd/data/2022/723656-23049-2022.gz This file has no extension and it isn't clear what format I am supposed to read it as.
2. https://noaaisd.blob.core.windows.net/noaa-isd/isd-lite/data/2022/723656-23049-2022.gz This file is also missing an extension, but appears to be a .tsv file with no headers, and while I was able to find some information about what the columns 'may' mean, from one of the technical documents, the data is only available for one month of 2022, when NOAA updates it daily. So it doesn't seem to have been updated since January.

Neither one of the files I found available from Planetary Computer match the hourly data files available directly from NOAA. Could the documentation be updated for this dataset to make it possible to navigate the storage blob, and to understand how this data differs from the data that is available directly from the source?

[TomAugspurger]

I think the PDF linked from the dataset page gives the information you need, but this seems like a pretty complicated format to parse.

@gadomski, who was working on the STAC metadata for this dataset before its priority was bumped down, put together pyisd to help parse the data, if you're using Python.

pip install -q git+https://github.com/gadomski/pyisd/

import isd.io
import urllib.request

filename, _ = urllib.request.urlretrieve("https://noaaisd.blob.core.windows.net/noaa-isd/data/2022/723656-23049-2022.gz", filename="/tmp/isd.gz")

df = isd.io.read_to_data_frame(filename)
df.head()

	usaf_id	ncei_id	year	month	day	hour	minute	data_source	latitude	longitude	report_type	elevation	call_letters	quality_control_process	wind_direction	wind_direction_quality_code	wind_observation_type	wind_speed	wind_speed_quality_code	ceiling	ceiling_quality_code	ceiling_determination_code	cavok_code	visibility	visibility_quality_code	visibility_variability_code	visibility_variability_quality_code	air_temperature	air_temperature_quality_code	dew_point_temperature	dew_point_temperature_quality_code	sea_level_pressure	sea_level_pressure_quality_code	additional_data	remarks	element_quality_data	original_observation_data	timestamp
0	723656	23049	2022	1	1	0	51	6	35.611	-106.095	FM-16	1934	KSAF	nan	330	5	N	1.5	5	640	5	M	N	16093	5	N	5	1	5	1	5	nan	9	GA1075+006405999GA2085+009755999GD13991+0064059GD24991+0097559MA1100145079155	MET10812/31/21 17:51:03 SPECI KSAF 010051Z 33003KT 10SM BKN021 OVC032 01/01 A2957 RMK AO2 RAB25E42 P0000 FIBI (SF)			2022-01-01 00:51:00
1	723656	23049	2022	1	1	0	53	7	35.611	-106.095	FM-15	1934	KSAF	nan	340	5	N	1.5	5	610	5	M	N	16093	5	N	5	1.1	5	0.6	5	1000.1	5	AA101000025AU110020015AW1615GA1075+006105999GA2085+009755999GD13991+0061059GD24991+0097559GE19AGL +99999+99999GF199999999999006101999999MA1100145079155MW1615	MET12112/31/21 17:53:03 METAR KSAF 010053Z 34003KT 10SM -RA BKN020 OVC032 01/01 A2957 RMK AO2 RAB25 SLP001 P0000 T00110006 (SF)			2022-01-01 00:53:00
2	723656	23049	2022	1	1	1	2	7	35.611	-106.095	FM-16	1934	KSAF	nan	100	5	N	1.5	5	945	5	M	N	16093	5	N	5	1.7	5	0.6	5	nan	9	AA101000021AU110020015AW1615GA1045+006105999GA2085+009455999GD12991+0061059GD24991+0094559GE19AGL +99999+99999GF199999999999006101999999MA1100145079155MW1615	MET10812/31/21 18:02:03 SPECI KSAF 010102Z 10003KT 10SM -RA SCT020 OVC031 02/01 A2957 RMK AO2 P0000 T00170006 (SF)			2022-01-01 01:02:00
3	723656	23049	2022	1	1	1	11	7	35.611	-106.095	FM-16	1934	KSAF	nan	130	5	N	2.1	5	610	5	M	N	16093	5	N	5	2.8	5	1.1	5	nan	9	AA101000021AU110020015AW1615GA1075+006105999GA2075+007925999GA3085+009455999GD13991+0061059GD23991+0079259GD34991+0094559GE19AGL +99999+99999GF199999999999006101999999MA1100145079155MW1615	MET11512/31/21 18:11:03 SPECI KSAF 010111Z 13004KT 10SM -RA BKN020 BKN026 OVC031 03/01 A2957 RMK AO2 P0000 T00280011 (SF)			2022-01-01 01:11:00
4	723656	23049	2022	1	1	1	48	7	35.611	-106.095	FM-16	1934	KSAF	nan	160	5	N	2.6	5	610	5	M	N	6437	5	N	5	3	5	2	5	nan	9	AA101001731AU120020015AU200001015AW1105AW2625GA1025+002745999GA2075+006105999GA3085+008235999GD11991+0027459GD23991+0061059GD34991+0082359GE19AGL +99999+99999GF199999999999002741999999MA1100105079125MW1655	MET10612/31/21 18:48:03 SPECI KSAF 010148Z 16005KT 4SM RA BR FEW009 BKN020 OVC027 03/02 A2956 RMK AO2 P0007 (SF)			2022-01-01 01:48:00

Neither one of the files I found available from Planetary Computer match the hourly data files available directly from NOAA. Could the documentation be updated for this dataset to make it possible to navigate the storage blob, and to understand how this data differs from the data that is available directly from the source?

The files under /data come directly from NOAA under the Open Data Dissemination Program, so those should be byte-for-byte identical to what is on the source's server. I'll have to double check, but we might have been pulling isd-lite separately (since we found the "main" ISD files pretty complicated, and wanted to offer something simpler to use).

Once we get back to this dataset we'll make sure the documentation is much improved.