Skip to content

Commit

Permalink
adding riverscapesXML and riverscapes.net content for migration
Browse files Browse the repository at this point in the history
  • Loading branch information
@MattReimer
MattReimer committed Dec 14, 2023
1 parent fae361e commit 24e7d1d
Show file tree
Hide file tree
Showing 36 changed files with 3,210 additions and 10 deletions.
23 changes: 23 additions & 0 deletions content/page/MIGRATED/About/index.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
title: About RC
description: About the Riverscapes Consortium
banner: true
---

## Overview of Riverscapes Consortium

In this less than five minute video we introduce the Riverscape Consortium.

<Youtube embedId="mfyhjVfECf4" />

## More Elaborate Background

This half hour video is based on [Joe Wheaton's](http://joewheaton.org) talk at a workshop in Lyon, France in December of 2019 on Regionalized Fluvial Geomorphology (see [original slides](https://www.researchgate.net/publication/338225952_Riverscapes_Consortium_to_Leverage_Advances_in_Other_Regions_in_your_Regionalization)). It lays out some of the key concepts and part of the motivation for the Riverscapes Consortium and why it we are developing data standards and workflows.

<Youtube embedId="OetscVdkr2I" />

## Talk about Need for Riverscapes Consortium and Standards

The Riverscape Consortium's [Joe Wheaton](http://joewheaton.org) was awarded the [British Society for Geomorphology's](https://twitter.com/BSG_Geomorph) 2020 [Gordon Warwick Award](https://www.geomorphology.org.uk/awards) for excellence in geomorphic research. In his talk below, he discusses the need for the sort of standards and community embodied in the Riverscape Consortium to rise to the challenge of the scope of riverscape degradation.

<Youtube embedId="EaLN3wZ6cM8" />
4 changes: 3 additions & 1 deletion content/page/MIGRATED/Data_Warehouses/gettingaround.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@ We show you how to navigate and view the Programs you have been granted access

### Access - Types

<RSStaticImage src="/images/data/Access.png" to="https://data.riverscapes.net/"/> All users have access to the "[Public Data](https://data.riverscapes.net/#/PUBLIC_DATA)". Each account has four different access levels defined in every Program:
<RSStaticImage src="/images/data/Access.png" to="https://data.riverscapes.net/" caption="caption" />

All users have access to the "[Public Data](https://data.riverscapes.net/#/PUBLIC_DATA)". Each account has four different access levels defined in every Program:

1. `No Access` - By default users do not have access to any other programs but the [Public Data](https://data.riverscapes.net/#/PUBLIC_DATA)
2. `Observer` - This provides basic read-only access to a Program.
Expand Down
80 changes: 80 additions & 0 deletions ...chnical_Reference/Documentation_Standards/Riverscapes_Projects/Program/index.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
title: Program XML
---

### Quick Link:

Validating your `program.xml` file can be done using the following URL:

```
https://raw.githubusercontent.com/Riverscapes/Program/master/Program/XSD/V1/Program.xsd
```

like so:

``` xml
<?xml version="1.0" encoding="utf-8"?>
<Program name="Riverscapes" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Riverscapes/Program/master/Program/XSD/V1/Program.xsd">
```

## Background:

We need a way to represent the bucket file structure and hold that structure
programatically so that products can find their homes when they upload and also
so users can find the products they care about.

The following is meant to completely describe the ideas of the following document:
<https://docs.google.com/spreadsheets/d/1PBTDZ_R_fdjydQ8jXVQr1Uf5OCOrzXhkfu9wSvvliIQ/>

So according to this, you would be able to find the output of a fuzzy FHM product like so:
`/ColumbiaRiverBasin/JohnDay/Sites/CBW05583-028079/VISIT_1029/2012/Q_001/Analyses/Fuzzy/Steelhead/Spawner/FHM/outputs/fuzzyHSI.tif`

We do this using some basic building blocks:

* `<Container name="Network"> `: A container is a folder literally called what the name is "Network" in this case
* `<Level name="Watershed"> `: A level will have repeating elements. So instead of "Watershed" you'll have lots
of folders, each wih the name of the Level (Watershed in this case) in question/
* `<Product>` : Products are the endpoint. If I can find a product then I can download or upload a whole project

```xml
<?xml version="1.0" encoding="utf-8"?>
<Program name="Riverscapes" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="Program.xsd">
<MetaData>
<Meta name="s3bucket">riverscapesdata</Meta>
</MetaData>
<Level name="Region">
<Level name="Watershed">
<Container name="Watershed">
<Product name="GRTS Rollup" id="GRTS" folder="grts"/>
<Product name="Context Layers" id="context" folder="context_layers"/>
</Container>
<Container name="Network">
<Product name="BRAT Models" id="BRAT" folder="brat"/>
<Product name="WRAT" id="WRAT" folder="wrat"/>
<Product name="Matt Imputation Crap" id="MIC" folder="matt_impute"/>
<Product name="GPP" id="GPP" folder="gpp"/>
<Product name="Capacity" id="capacity" folder="capacity"/>
<Product name="River Styles" id="RSTYLES" folder="riverstyles"/>
</Container>
<Container name="Sites">
<Level name="Site">
<Level name="Visit">
<Level name="Year">
<Level name="Flow">
<Product name="FHM" id="FHM" folder="FHM"/>
<Product name="GUT" id="GUT" folder="GUT"/>
<Product name="GCD" id="GCD" folder="GCD"/>
</Level>
<Container name="Topography">
<Product name="DEM" id="DEM" folder="DEM"/>
</Container>
</Level>
</Level>
</Level>
</Container>
</Level>
</Level>
</Program>
```
136 changes: 136 additions & 0 deletions ...l_Reference/Documentation_Standards/Riverscapes_Projects/Project/datasources.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
---
title: DataSources (Inputs/Outputs)
weight: 3

---

DataSources can be defined in two places:

1. **Inside** **`<Inputs>` At the top of the file**: These are DataSources that can be referenced from other places.
2. **Inside a realization**: This can either be a reference to the input at the top of the file or a completely new DataSource node.


## Dataset Type Hierarchy

The kind of Datasource tag you choose (e.g. `<Raster>`, `<DEM>` ) is important. The list below may not be extensive since new tags may be invented from time to time.

1. **DataSet**: *This is a grouping only and not in use as a tag*
1. **GeoSpatial**: *This is a grouping only and not in use as a tag*
1. **Raster**: `<Raster id="" Guid="">`
1. **DEM**: `<DEM id="" Guid="">`
2. **Vector**: `<Vector id="" Guid="">`
3. **TIN**: `<TIN id="" Guid="">`
2. **Context Layer**: A context layer can either be a file OR a tile-layer loaded from a mapping service
1. **Hillshade**: `<Hillshade>`: Path to a hillshade raster or tile service.
2. **Shape**: A `.shp` file that is not used for context and not geospatial processing.
3. **SimpleFile**
1. **CSV**: Self-explanatory
2. **Image**: Any image (png, jpg, gif etc.)

## General Use

**Here is an example of a DataSource**

```xml
<Vector id="hillshade01" Guid="241cdf2a-a397-4fd7-acd2-de0869ed4662">
<Name>Hillshade</Name>
<Path>analyses\context\layers\hillshade\Upper_Salmon_Hillshade.tif</Path>
<Project>../../Relative/path/to/OtherProject</Project>
</Vector>
```

### Attributes:

* **Guids**: `Guid=""` All inputs that are not references (i.e. using the `ref=""` attribute) must have a Guid.
* **Id**: `id=""` The Id you give to a Datasource should be unique in the XML file but does not have to be unique across all project XMLs.
* **References**: `ref=""`: The `ref` attribute is used to reference a DataSource inside the `<Inputs>` tag at the top of the file. It should contain the unique `id` of the DataSource you would like to reference.

#### Rerences (`ref=""`)

**Some rules for using `ref`**

* If you use `ref` you should not use `Id` or `Guid` since these are covered inside the referenced object. Likewise you should not use `<Path>`, `<Project>`, `<Name>` or any other tags.
* `<MetaData>` can still be used but only if it is refering to information that cannot be shared across all [Realizations]().

``` xml
<Project>
<!-- ... -->
<Inputs>
<!-- Here we define the actual input -->
<DEM id="DEM1" Guid="241cdf2a-a397-4fd7-acd2-de0869ed4662">
<Name>DEM</Name>
<Path>01_Inputs/01_Topo/Slope/Slope1.img</Path>
<MetaData>
<Meta Name="description">Jordan Gilbert</Meta>
<Meta Name="HUC8ID">17020005</Meta>
</MetaData>
</DEM>
</Inputs>
<Realizations>
<VBET>
<!-- ... -->
<Inputs>
<Topography>
<!-- And here we refer to it -->
<DEM ref="DEM1" />
</Topography>
</Inputs>
<!-- ... -->
</VBET>
</Realizations>
</Project>
```

### Elements:

* **Name**: `<Name>` A plain-text name for this Datasource
* **Path**: `<Path>` A ***RELATIVE*** filepath to the layer/file/datasource in question (i.e. `inputs/input01/input.tif`). The path is relative either to this project or another project root specified using the `<Project>` element.
* **Project**: `<Project>`*(Optional)* If you are refering to a datasource in another project you can use this element. Relative paths are preferred but in some cases absolute paths are necessary. See below for more details.
* [**MetaData**](metadata.html): `<MetaData>` Key-value pairs. *(See [MetaData](metadata.html) docs for how to use this.)*

#### Special Elements:

* **URL**: *(Contextual Layers Only)* Contextual layers like `<Hillshade>` can have this element. It is the path to an online location or tile-server. The idea is that the software can try to load the local file and, failing this, use the version available from the `<URL>`

## External Project References

You can use Datasources from other projects using the `<Project>` element

* Relative paths are prefered. Absolute paths can be used but will break when transfered to another machine.
* You *must* specify a path to a FOLDER contianing the `project.rs.xml` file: `../../Path/to/Project/`

## Extended Use

Sometimes you'll see a tag inside an `<Analysis>` tag that is Dataset-like. An example is VBET where a drainage network `<Network>` is basically a shapefile Dataset Type that has been extended to contain `<Buffers>`

```xml
<Inputs>
<Topography>
<DEM ref="DEM1"/>
<Flow ref="FLOW4235"/>
<Slope ref="SLOPE4235"/>
</Topography>
<DrainageNetworks>
<!-- Network is an extended DataSet that has been augmented with <Buffers> -->
<Network ref="DN001">
<Buffers>
<!-- A buffer is simply <Vector> object renamed -->
<Buffer>
<Name>Large Buffer</Name>
<Path>01_Inputs/02_Network/Network_001/Buffers/lg_buf1.shp</Path>
</Buffer>
<Buffer>
<Name>Medium Buffer</Name>
<Path>01_Inputs/02_Network/Network_001/Buffers/med_buf1.shp</Path>
</Buffer>
<Buffer>
<Name>Small Buffer</Name>
<Path>01_Inputs/02_Network/Network_001/Buffers/sm_buf1.shp</Path>
</Buffer>
</Buffers>
</Network>
</DrainageNetworks>
</Inputs>
```

Such examples are too specific to document fully here. You can find hints at how to use these using autocomplete in Pycharm or by reading the XSD file.
94 changes: 94 additions & 0 deletions ...chnical_Reference/Documentation_Standards/Riverscapes_Projects/Project/guids.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
---
title: GUIDs
weight: 1
---

[Universally Unique Identifiers (UUID, or GUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier) Are used to track file changes on XML nodes.

```
07817542-0b3f-4154-9bfb-d3e20cc69d64
740ce55d-91f7-47bd-b3e7-265d685a464d
174fd928-34a1-462f-ae60-8a6d3109c97b
c767a4b0-74db-49d2-bfbf-8b6d3a77b3d5
603ee6e2-a70d-4514-adf0-7335b9800743
b5b5bb74-7dbd-468d-a863-a026b36a0224
385c98ec-811c-4dd0-8990-3b92c33f3f5b
1f819399-1f97-4cec-a8d7-b5180aca6221
f0b50bdc-e302-4647-ae04-98f561f90b38
412332d4-89a0-4acb-a7ae-2826f8717b55
```

## Guidelines for GUID use

### DO use GUIDs:

* Uniquely. Every GUID inside an XML file should be unique. There is one case where GUIDs can be duplicates: when you are referencing a layer from another project.
* On every file node for inputs and outputs: (`<Raster>, <Vector> etc...`)
* On Every Realization
* On inputs pulled from other projects. **Note: GUIDs must match.**

### DON'T use GUIDS:

* On container nodes (`<Outputs></Outputs>`)
* On internal references: (`<Raster ref="MYRASTER"/>`)

### DO Change a GUID

* Any time the file on the disk is altered in any way.

## Examples:

``` xml
<!-- Note the Raster and not the container has the GUID -->
<Inputs>
<Raster id="SLOPE4235" guid="241cdf2a-a397-4fd7-acd2-de0869ed4662">
<Name>Slope</Name>
<Path>01_Inputs/01_Topo/Slope/Slope1.img</Path>
</Raster>
</Inputs>


<!-- Here's a realization example -->
<VBET dateCreated="2016-07-08T23:49:51" productVersion="1.4.0" guid="385c98ec-811c-4dd0-8990-3b92c33f3f5b">
...
</VBET>


<!-- This guid should match the one in the VBET project -->
<Vector id="valleybottom01" guid="b5b5bb74-7dbd-468d-a863-a026b36a0224">
<Name>USal_valley_bottom</Name>
<Path>inputs/USal_valleybottom.shp</Path>
<Project>../some/../other/path/Sample_VBET_Project/</Project>
</Vector>

```



## Code Examples:

### Python:

```python
import uuid

def getUUID(self):
"""
Here's a little function that prints a unique ID (GUID)
"""
return str(uuid.uuid4()).upper()

print getUUID()
```

Note that this uses `.upper()` to create windows-like GUIDs that are all uppercase. It shouldn't matter that much whether you create upper or lowercase GUIDs so long as you are consistent.



### Fetching Guids

When you're linking projects together you might need to fetch a guid from anothe project.

The following snippet shows 3 different ways to retrieve a guid from a project:

<https://github.com/Riverscapes/Program/blob/master/Project/python/guid.py>
Loading

0 comments on commit 24e7d1d

Please sign in to comment.