generated from jtr13/quarto-edav-template
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ch6_a.qmd
405 lines (270 loc) · 17.3 KB
/
ch6_a.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
---
title-block-banner: true
bibliography: references.bib
editor:
markdown:
wrap: sentence
---
## Popular Climate Data Format
Raster and netCDF are two popular formats used for gridded climate data dissemination and archiving.
We are all too familiar with the raster format (pixelated, georeferenced data) from the previous chapters.
NetCDF (Network Common Data Form), is a common data type for multi-layered, structured, gridded dataset.
NetCDF is a machine-independent data format and is a community standard for sharing scientific data.
A netCDF has certain features which makes it suitable for complex scientific data archiving and sharing, namely,
<div>
> - *Self-Describing*.
> A netCDF file includes information about the data it contains.
>
> - *Portable*.
> A netCDF file can be accessed by computers with different ways of storing integers, characters, and floating-point numbers.
>
> - *Scalable*.
> A small subset of a large dataset may be accessed efficiently.
>
> - *Appendable*.
> Data may be appended to a properly structured netCDF file without copying the dataset or redefining its structure.
>
> - *Shareable*.
> One writer and multiple readers may simultaneously access the same netCDF file.
>
> - *Archivable*.
> Access to all earlier forms of netCDF data will be supported by current and future versions of the software.
>
> \- From \[Unidata \| NetCDF (ucar.edu) <https://www.unidata.ucar.edu/software/netcdf/>\]
</div>
{width="750"}
Several open-source plaforms and agencies provide open access to a multitude of gridded land and climate datasets generated using satellites and land-surface/ climate models.
In the following sections, we will familiarize ourselves with some of these resources.
## Open-Data Platforms
### Climate Data from NOAA Physical Sciences Lab
- Product overview / Data access/ Information: <https://psl.noaa.gov/data/gridded/tables/daily.html>
[{width="750"}](https://psl.noaa.gov/data/gridded/tables/daily.html)
This website provides several land and climate variables such as: CPC Global Unified Gauge-Based Analysis of Daily Precipitation, CPC Global Temperature, NCEP/NCAR Reanalysis, Livneh daily CONUS near-surface gridded meteorological and derived hydrometeorological data.
### DAYMET: Daily Gridded Weather and Climate Data for North America \[1 km x 1 km\]
- Product overview: <https://daymet.ornl.gov/>
- Data access: <https://daac.ornl.gov/cgi-bin/dataset_lister.pl?p=32>
- Data information: Refer to the User Guide provided with each dataset.
{width="750"}
Daymet provides long-term, continuous, gridded estimates of daily weather and climatology variables at 1 km grid resolution for North America.
The dataset is available in several forms, including monthly and annual climate summaries, in addition to the daily and/or sub-daily climate forcings:
1. [Sub-daily Climate Forcings for Puerto Rico](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1977)
2. [Daymet Version 4 Monthly Latency: Daily Surface Weather Data](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1904)
3. [Annual Climate Summaries on a 1-km Grid for North America, Version 4 R1](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2130)
4. [Monthly Climate Summaries on a 1-km Grid for North America, Version 4 R1](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2131)
5. [Station-Level Inputs and Cross-Validation for North America, Version 4 R1](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2132)
6. [Daily Surface Weather Data on a 1-km Grid for North America, Version 4 R1](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2129)
[{width="450"}](https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2129)
### CHIRPS Global Precipitation \[\~5 km x 5km\]
- Product overview: <https://climatedataguide.ucar.edu/climate-data/chirps-climate-hazards-infrared-precipitation-station-data-version-2>
- Data access: <https://data.chc.ucsb.edu/products/CHIRPS-2.0/>
- Data information: [https://data.chc.ucsb.edu/products/CHIRPS-2.0/docs/README-CHIRPS.txt](#0)
[{width="450"}](https://data.chc.ucsb.edu/products/CHIRPS-2.0/)
### GIMMS MODIS Global NDVI \[\~225 m x 225 m\]
- Product overview: <https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/products/MOD13Q1>
- Data access: <https://gimms.gsfc.nasa.gov/MODIS/>
- Data information: <https://gimms.gsfc.nasa.gov/MODIS/README-global.txt>
[{width="750"}](https://gimms.gsfc.nasa.gov/MODIS/)
### Climate Prediction Center
- Product overview / Data access/ Information: <https://ftp.cpc.ncep.noaa.gov/GIS/>
[{width="400"}](https://ftp.cpc.ncep.noaa.gov/GIS/)
Includes several variables including (but not limited to):
- Climate Prediction Center (CPC) Morphing Technique (MORPH) to form a global, high resolution precipitation analysis
- Joint Agricultural Weather Facility (JAWF)
- Grid Analysis and Display System (GRADS): Global precipitation monitoring and forecasts, Tmax, Tmin
- Input variables for US Drought Monitor (USDM)
### Soil Texture for CONUS \[30 m x 30 m\]
Probabilistic Remapping of SSURGO (POLARIS) is a database of 30-m probabilistic soil property maps over the Contiguous United States (CONUS) generated by removing artificial discontinuities in Soil Survey Geographic (SSURGO) database.
using an artificial intelligence algorithm [@chaney2016polaris; @chaney2019polaris].
Estimates provided by POLARIS include soil texture, organic matter, pH, saturated hydraulic conductivity, Brooks-Corey and Van Genuchten water retention curve parameters, bulk density, and saturated water content for six profile depths, namely, 0-5 cm, 5-15 cm, 15-30 cm, 30-60 cm, 60-100 cm, 100-200 cm.
- Product overview: <https://www.usgs.gov/publications/polaris-properties-30-meter-probabilistic-maps-soil-properties-over-contiguous-united>
- Data access: <http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/>
- Data information: <http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/Readme>
[{width="450"}](http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/)
### **NASA *A****ρρ***EEARS**
- Product overview / Data access/ Information: <https://appeears.earthdatacloud.nasa.gov/>
[{width="750"}](https://appeears.earthdatacloud.nasa.gov/)
Climate/ land data variables can be extracted for an area or a point using the interactive interface.
Click on point samples/ area samples:
{width="400"}
{width="400"}
### NASA Earth Data Search
- Product overview / Data access/ Information: <https://search.earthdata.nasa.gov/search>
[{width="750"}](https://search.earthdata.nasa.gov/search)
#### Bulk Download Order
NASA Earth Data provides customization options for bulk data download.
Lets say we are interested in downloading global SMAP Level 3 soil moisture.
We start by selecting the product, and specify start/ end date as needed.
1. <div>
{width="750"}
</div>
2. <div>
{width="750"}
</div>
3. <div>
{width="750"}
</div>
4. <div>
{width="750"}
</div>
5. <div>
{width="500"}
</div>
6. <div>
{width="500"}
</div>
Copy and Paste these links in any internet download manager (my favorite in Chrono for Google Chrome), Select output location (typically an external hard drive) and let the download begin.
## Programmatic Data Acquisition
In the HTTP, FTP or HTP links provided before, one can download a file by clicking on the individual hyperlink.
Alternatively, we can use `download.file` function to download the file programmatically in R.
This help us by opening the path to automate download and processing of multiple files with minimal supervision.
### Downloading Raster files
Let us take an example of `us_tmax` data available at: [https://ftp.cpc.ncep.noaa.gov/GIS/GRADS_GIS/GeoTIFF/TEMP/us_tmax/]{.underline}
Right-click on the raster file for 20240218, and copy the file path.
We will then use this link to access the files programmatically using *Client URL*, or *cURL* - a utility for transferring data between systems.
We will download the raster using `download.file` to local disk, and saved with a uder-defined name `tmax_20240218.tif`.
{width="450"}
Copied link: [https://ftp.cpc.ncep.noaa.gov/GIS/GRADS_GIS/GeoTIFF/TEMP/us_tmax/us.tmax_nohads_ll_20240218_float.tif]{.underline}
```{r 6a1, message = FALSE, warning = FALSE}
# Copied path of the raster
data_path <- "https://ftp.cpc.ncep.noaa.gov/GIS/GRADS_GIS/GeoTIFF/TEMP/us_tmax/us.tmax_nohads_ll_20240218_float.tif"
# Download the raster using download.file, assign the name tmax_20240218.tif to the downloaded
download.file(url = data_path,
method="curl",
destfile = "tmax_20240218.tif")
# Plot downloaded file
library(terra)
tempRas=rast("tmax_20240218.tif") # Import raster to the environment
usSHP=terra::vect(spData::us_states) # Shapefile for CONUS
plot(tempRas)
plot(usSHP, add=TRUE)
```
Now that we have the `tmax_20240218` raster, let us extract the values for certain selected locations: s
```{r 6a2, message = FALSE, warning = FALSE}
# Import sample locations from contrasting hydroclimate
library(readxl)
loc= read_excel("./SampleData-master/location_points.xlsx")
print(loc)
# Value of the lat & lon of the locations
latlon=loc[,3:4]
print(latlon)
# Extract time series using "terra::extract"
loc_temp=terra::extract(tempRas,
latlon, #2-column matrix or data.frame with lat-long
method='bilinear') # Use bilinear interpolation (or ngb) option
print(loc_temp)
# Add temperature attribute to the data frame as "temp"
loc$temp=loc_temp$tmax_20240218
print(loc)
# Export the modified data as CSV
write.csv(loc, "df_with_temp.csv")
```
### Raster Mosaic
Lets explore raster files for mean clay percentage for top 5 cm soil profile accessible through: <http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/clay/mean/0_5/>.
The link provides rasters for 1x1 degree areal domain.
```{r 6a3, message = FALSE, warning = FALSE}
library(spData)
# Note the spatial extent of Louisiana State. We need corresponding raster files
ext(spData::us_states[spData::us_states$NAME=="Louisiana",]) # ext function gives extent of the rast/vect object
pth1="http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/clay/mean/0_5/lat3031_lon-91-90.tif"
# Download the raster using download.file
download.file(url = pth1,
method="curl",
destfile = "lat3031_lon-90-89.tif")
# Import the downloaded raster to workspace
r1=rast("lat3031_lon-90-89.tif")
# Fetch shapefile for Louisiana from spData package
LAvct=vect(spData::us_states[spData::us_states$NAME=="Louisiana",])
# Plot downloaded raster against the map of Louisiana. Note that the raster fits 1x1 grid
plot(LAvct, col="gray90")
plot(r1, range=c(0,100), add=TRUE)
grid()
# Crop raster to smaller region of interest for easy visualization
#~~~ We will crop raster to a user-defined extent
r1crp=crop(r1, ext(c(-91,-90.8,30,30.2)))
# Explore a smaller subset of the dataset
#~~~ Can you identify the Mississipi flood-plain using the clay percentage?
library(mapview)
mapview(r1crp, # Raster to be plotted
at=c(0,10,20,30,40,50,60,75), # Legend breaks
map.types="Esri.WorldImagery", # Select background map
main="Clay % (0-5 cm profile)") # Plot title
```
We note that the raster files in POLARIS are available for a 1x1 areal domain.
For an analysis for a large spatial extent, multiple smaller rasters can be "stitched" together to generate a larger mosaic of rasters.
We will use the `terra::mosaic` function to generate a mosaic of several smaller rasters of percentage clay content in 0-5 cm soil profile in Southeastern Louisiana.
This function requires the user to specify the summary function ("sum", "mean", "median", "min", or "max") to be applied on the overlapping pixels from 2 or more rasters.
Let us download some more rasters from POLARIS and *mosaic* them together.
{width="300"}
```{r 6a4, message = FALSE, warning = FALSE}
# Links to soil texture rasters
pth2="http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/clay/mean/0_5/lat2930_lon-91-90.tif"
pth3="http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/clay/mean/0_5/lat3031_lon-90-89.tif"
pth4="http://hydrology.cee.duke.edu/POLARIS/PROPERTIES/v1.0/clay/mean/0_5/lat3031_lon-92-91.tif"
# Download the raster using download.file function
download.file(url = pth2,
method="curl",
destfile = "r2.tif")
download.file(url = pth3,
method="curl",
destfile = "r3.tif")
download.file(url = pth4,
method="curl",
destfile = "r4.tif")
# Import downloaded files to workspace
r2=rast("r2.tif")
r3=rast("r3.tif")
r4=rast("r4.tif")
# DIY: Plot the downloaded raster against the map of Louisiana
# plot(LAvct)
# plot(r1, range=c(0,100), add=TRUE)
# plot(r2, range=c(0,100), add=TRUE, legend=FALSE)
# plot(r3, range=c(0,100), add=TRUE, legend=FALSE)
# plot(r4, range=c(0,100), add=TRUE, legend=FALSE)
# Raster mosaic
r_mos=mosaic(r1,r2,r3,r4, fun="mean")
# Plot raster mossaic
plot(LAvct, main="Clay % [0-5 cm depth]",axes=FALSE, col="gray90")
plot(r_mos, range=c(0,100), add=TRUE)
plot(LAvct, add=TRUE)
grid()
```
For easy post processing access, its a good practice to save the processed dataset on disc as either a raster or netCDF.
```{r 6a4b, message = FALSE, warning = FALSE}
# Export raster to disc
terra::writeRaster(r_mos, "clayPct.tif",
overwrite=TRUE) # Overwrite if the file already exists?
# Or export as netCDF
terra::writeCDF(r_mos, "clayPct.nc",
overwrite=TRUE) # Overwrite if the file already exists?
# Optional: Add more information to the exported netCDF
# terra::writeCDF(r_mos, # SpatRaster to export
# "clayPct.nc", # Output filename
# varname="clayPCT", # Short name of the variable
# unit="%", # Variable units
# longname="Clay Percentage, 0-5 cm, POLARIS", # Long name of the variable
# zname='[-]') # Z-var name (None, since we export a single layer)
```
### Downloading netCDF
We will now download a netCDF of global daily precipitation for the year 2023 from CHIRPS, accessible through the link: [https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p05/chirps-v2.0.2023.days_p05.nc]{.underline}
```{r 6a5, message = FALSE, warning = FALSE}
# Copied path of the raster
data_path <- "https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p05/chirps-v2.0.2023.days_p05.nc"
# Download the raster using download.file, assign the name "daily_pcp_2023.nc" to the downloaded
if (file.exists("daily_pcp_2023.nc")==FALSE){
download.file(url = data_path,
method="curl",
destfile = "daily_pcp_2023.nc")
}
# Plot downloaded file
library(terra)
pcp=rast("daily_pcp_2023.nc") # Import raster to the environment
print(pcp) # Notice the attributes (esp. nlyr, i.e. number of layers, unit and time)
head(time(pcp)) # time variable in the netCDF indicating corresponding time of acquisition
worldSHP=terra::vect(spData::world) # Shapefile for CONUS
# Plot data for a specific layer
plot(pcp[[100]]) # Same as pcp[[which(time(pcp)=="2023-04-10")]]
plot(worldSHP, add=TRUE)
points(latlon, pch=19, col="red")
```
------------------------------------------------------------------------
{fig-align="center" width="600"}