geofabrics.processor

This module contains classes associated with generating GeoFabric layers from LiDAR and bathymetry contours based on the instructions contained in a JSON file.

GeoFabric layers include hydrologically conditioned DEMs.

Classes

BaseProcessor	An abstract class with general methods for accessing elements in
RawLidarDemGenerator	RawLidarDemGenerator executes a pipeline for creating a DEM from LiDAR and
HydrologicDemGenerator	HydrologicDemGenerator executes a pipeline for loading in a DEM then
PatchDemGenerator	PatchDemGenerator executes a pipeline for loading in a DEM / roughness
RoughnessLengthGenerator	RoughnessLengthGenerator executes a pipeline for loading in a hydrologically
MeasuredRiverGenerator	MeasuredRiverGenerator executes a pipeline to interpolate between
RiverBathymetryGenerator	RiverbathymetryGenerator executes a pipeline to estimate river
WaterwayBedElevationEstimator	WaterwayBedElevationGenerator executes a pipeline to pull in OpenStreetMap waterway
StopbankCrestElevationEstimator	StopbankCrestElevationEstimator executes a pipeline to estimate stopbank

Module Contents

class geofabrics.processor.BaseProcessor(json_instructions)

Bases: abc.ABC

An abstract class with general methods for accessing elements in instruction files including populating default values. Also contains functions for downloading remote data using geopais, and constructing data file lists.

The BaseProcessor class contains several important class members:

• instructions - Defines the pipeline execution instructions

• catchment_geometry - Defines all relevant regions in a catchment required in the generation of a DEM as polygons.

• logger - logging within this class

Parameters:

json_instructions (json)

create_metadata()

A clase to create metadata to be added as netCDF attributes.

Return type:

dict

get_instruction_path(key, defaults={})

Return the file path from the instruction file, or default if there is a default value and the local cache is specified. Raise an error if the key is not in the instructions.

Parameters:

• key (str)

• defaults (dict)

Return type:

pathlib.Path

check_instruction_path(key)

Return True if the file path exists in the instruction file, or True if there is a default value and the local cache is specified.

Parameters:

key (str)

Return type:

bool

create_results_folder()

Ensure the results folder has been created.

save_dem(filename, dataset, generator, compression)

Save out the dem/geofabrics labelled array.

Parameters:

• filename (pathlib.Path) – The filename including extension to save the DEM/geofabric to.

• dataset (xarray.Dataset) – The labelled array data to save.

• generator (geofabrics.dem.DemBase) – The dem.DemBase object with a ‘save_dem’ function.

• compression (int)

get_resolution()

Return the resolution from the instruction file. Raise an error if not in the instructions.

Return type:

float

get_crs()

Return the CRS projection information (horiztonal and vertical) from the instruction file. Raise an error if ‘output’ is not in the instructions. If no ‘crs’ or ‘horizontal’ or ‘vertical’ values are specified then use the default value for each one missing from the instructions. If the default is used it is added to the instructions.

Return type:

dict

get_instruction_general(key, subkey=None)

Return the general instruction from the instruction file or return the default value if not specified in the instruction file. Raise an error if the key is not in the instructions and there is no default value. If the default is used it is added to the instructions.

Parameters:

• key (str)

• subkey (str)

get_processing_instructions(key)

Return the processing instruction from the instruction file or return the default value if not specified in the instruction file. If the default is used it is added to the instructions.

Parameters:

key (str) – The string identifying the instruction

check_datasets(key, data_type)

Check to see if the dataset is included in the instructions: key = dataservice (i.e. local, opentogaphy, linxz, lris)

Parameters:

• key (str) – The string identifying how the data is accessed (e.g. local, opentopography, linz, etc)

• data_type (str) – The string identifying the data type (e.g. raster, lidar, vector)

Return type:

bool

check_vector_or_raster(key, api_type)

Check to see if vector or raster key (i.e. land, ocean_contours, etc) is included either as a file path, or within any of API’s (i.e. LINZ or LRIS).

Parameters:

• key (str) – The string identifying the vector/raster

• api_type (str) – The string identifying if the key is a vector or raster

Return type:

bool

get_vector_or_raster_paths(key, data_type, required=True)

Get the path to the vector/raster key data included either as a file path or as an API. Return all paths where the vector key is specified. In the case that an API is specified ensure the data is fetched as well.

Parameters:

• key (str) – The string identifying the vector/raster

• data_type (str) – The string identifying if the key is a vector or raster

• required (bool) – If True, an exception will be raised if no path(s) are found.

Return type:

list

get_lidar_dataset_crs(data_service, dataset_name)

Checks to see if source CRS of an associated LiDAR dataset has be specified in the instruction file. If it has been specified, this CRS is returned, and will later be used to override the CRS encoded in the LAS files.

Return type:

dict

get_lidar_datasets_info()

Return a dictionary with three enties ‘file_paths’, ‘crs’ and ‘tile_index_file’. The ‘file_paths’ contains a list of LiDAR tiles to process.

The ‘crs’ (or coordinate system of the LiDAR data as defined by an EPSG code) is only optionally set (if unset the value is None). The ‘crs’ should only be set if the CRS information is not correctly encoded in this is only supported for OpenTopography LiDAR.

The ‘tile_index_file’ is also optional (if unset the value is None). The ‘tile_index_file’ should be given if a tile index file exists for the LiDAR files specifying the extents of each tile. This is currently only supported for OpenTopography files.

If a LiDAR dataset (either through an API or locally) is specified this is checked and all files within the catchment area are downloaded and used to construct the file list. If none is specified, the instruction ‘data_paths’ is checked for ‘lidars’ and these are returned.

Return type:

dict

abstract run()

This method controls the processor execution and code-flow.

class geofabrics.processor.RawLidarDemGenerator(json_instructions, debug=True)

Bases: BaseProcessor

RawLidarDemGenerator executes a pipeline for creating a DEM from LiDAR and optionally a coarse DEM. The data sources and pipeline logic is defined in the json_instructions file.

See the GitHub repository wiki or GeoFabrics/tests/ for usage examples and other documentation.

Parameters:

• json_instructions (json)

• debug (bool)

run()

This method executes the geofabrics generation pipeline to produce geofabric derivatives.

Note it currently only considers one LiDAR dataset that can have many tiles. See ‘get_lidar_datasets_info’ for where to change this.

class geofabrics.processor.HydrologicDemGenerator(json_instructions, debug=True)

Bases: BaseProcessor

HydrologicDemGenerator executes a pipeline for loading in a DEM then incorporating hydrological features (e.g. ocean bathymetry, rivers, waterways,stopbank creast heights) to produce a hydrologically conditioned DEM. The data and pipeline logic is defined in the json_instructions file.

See the GitHub repository wiki or GeoFabrics/tests/ for usage examples and other documentation.

Parameters:

• json_instructions (json)

• debug (bool)

add_hydrological_features(hydrologic_dem, area_threshold, catchment_dirs, temp_folder)

Add in any bathymetry data - ocean or river

Parameters:

• hydrologic_dem (geofabrics.dem.HydrologicallyConditionedDem)

• area_threshold (float)

• catchment_dirs (pathlib.Path)

• temp_folder (pathlib.Path)

run()

This method executes the geofabrics generation pipeline to produce geofabric derivatives.

class geofabrics.processor.PatchDemGenerator(json_instructions, debug=True)

Bases: BaseProcessor

PatchDemGenerator executes a pipeline for loading in a DEM / roughness GeoFabrics profuct before adding specified patches to it.

See the README.md for usage examples or GeoFabrics/tests/ for examples of usage and an instruction file

Parameters:

• json_instructions (json)

• debug (bool)

get_patch_instruction(key)

Return true if the DEMs are required for later processing

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

run()

This method executes the geofabrics generation pipeline to produce geofabric derivatives.

class geofabrics.processor.RoughnessLengthGenerator(json_instructions, debug=True)

Bases: BaseProcessor

RoughnessLengthGenerator executes a pipeline for loading in a hydrologically conditioned DEM and LiDAR tiles to produce a roughness length layer that is added to the Hydrologically conditioned DEM. The data and pipeline logic is defined in the json_instructions file.

The RoughnessLengthGenerator class contains several important class members:

• catchment_geometry - Defines all relevant regions in a catchment required in the generation of a DEM as polygons.

• roughness_dem - Adds a roughness layer to the hydrologically conditioned DEM.

Parameters:

• json_instructions (json)

• debug (bool)

get_roughness_instruction(key)

Return true if the DEMs are required for later processing

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

load_roads_osm()

Download OpenStreetMap roads within the catchment BBox.

Return type:

bool

run()

This method executes the geofabrics generation pipeline to produce geofabric derivatives.

class geofabrics.processor.MeasuredRiverGenerator(json_instructions, debug=True)

Bases: BaseProcessor

MeasuredRiverGenerator executes a pipeline to interpolate between measured river cross section elevations. A json_instructions file defines the pipeline logic and data.

Parameters:

• json_instructions (json)

• debug (bool)

get_measured_instruction(key)

Return true if the DEMs are required for later processing

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

estimate_river_mouth_fan(defaults)

Calculate and save depth estimates along the river mouth fan.

Parameters:

defaults (dict)

run()

This method extracts a main channel then executes the DemGeneration pipeline to produce a DEM before sampling this to extimate width, slope and eventually depth.

class geofabrics.processor.RiverBathymetryGenerator(json_instructions, debug=True)

Bases: BaseProcessor

RiverbathymetryGenerator executes a pipeline to estimate river bathymetry depths from flows, slopes, friction and widths along a main channel. This is dones by first creating a hydrologically conditioned DEM of the channel. A json_instructions file defines the pipeline logic and data.

Parameters:

• json_instructions (json)

• debug (bool)

channel_polyline  The main channel along which to estimate depth. This

is a polyline.

gen_dem  The ground DEM generated along the main channel. This is a

raster.

veg_dem  The vegetation DEM generated along the main channel. This is a

raster.

aligned_channel_polyline  The main channel after its alignment has been

updated based on the DEM. Width and slope are estimated on this.

transects  Transect polylines perpindicular to the aligned channel with

samples of the DEM values

channel_characteristics_exist()

Return true if the DEMs are required for later processing

Return type:

bool

channel_bathymetry_exist()

Return true if the river channel and bathymetry files exist.

Return type:

bool

alignment_exists()

Return true if the DEMs are required for later processing

Parameters:

behaviour (instructions The json instructions defining the)

Return type:

bool

get_result_file_name(key=None, name=None)

Return the file name of the file to save.

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

• name (str)

Return type:

str

get_result_file_path(key=None, name=None)

Return the file name of the file to save with the local cache path.

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

• name (str)

Return type:

pathlib.Path

get_bathymetry_instruction(key)

Return true if the DEMs are required for later processing

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

get_network_channel()

Read in or create a channel from a river network.

Return type:

geofabrics.bathymetry_estimation.Channel

get_dems(channel)

Allow selection of the ground or vegetation DEM, and either create or load it.

Parameters:

• behaviour (instructions The json instructions defining the)

• channel (geofabrics.geometry.CatchmentGeometry)

Return type:

tuple

align_channel(channel_width, channel)

Align the river network defined channel based on LiDAR and save the aligned channel.

Parameters:

• other (channel_width The class for characterising channel width and) – properties

• alignment (channel The river network defined channel)

• channel_width (geofabrics.bathymetry_estimation.ChannelCharacteristics)

• channel (geofabrics.bathymetry_estimation.Channel)

Return type:

geopandas.GeoDataFrame

calculate_channel_characteristics(channel_width, aligned_channel)

Align the river network defined channel based on LiDAR and save the aligned channel.

Parameters:

• other (channel_width The class for characterising channel width and) – properties

• alignment (channel The river network defined channel)

• channel_width (geofabrics.bathymetry_estimation.ChannelCharacteristics)

• aligned_channel (geopandas.GeoDataFrame)

Return type:

tuple

characterise_channel()

Calculate the channel width, slope and other characteristics. This requires a ground and vegetation DEM. This also may require alignment of the channel centreline.

align_channel_from_rec()

Calculate the channel width, slope and other characteristics. This requires a ground and vegetation DEM. This also may require alignment of the channel centreline.

Return type:

tuple

align_channel_from_osm()

Calculate the channel width, slope and other characteristics. This requires a ground and vegetation DEM. This also may require alignment of the channel centreline.

Return type:

geofabrics.bathymetry_estimation.ChannelCharacteristics

calculate_river_bed_elevations()

Calculate and save depth estimates along the channel using various approaches.

estimate_river_mouth_fan()

Calculate and save depth estimates along the river mouth fan.

run()

This method extracts a main channel then executes the DemGeneration pipeline to produce a DEM before sampling this to extimate width, slope and eventually depth.

class geofabrics.processor.WaterwayBedElevationEstimator(json_instructions, debug=True)

Bases: BaseProcessor

WaterwayBedElevationGenerator executes a pipeline to pull in OpenStreetMap waterway and tunnel information. A DEM is generated of the surrounding area and this used to unblock waterways and tunnels - by taking the lowest value in the area around a tunnel to be the tunnel elevation, and ensuring opne waterways flow downhill.

Parameters:

• json_instructions (json)

• debug (bool)

load_dem(filename)

Load a DEM

Parameters:

filename (pathlib.Path)

get_waterways_instruction(key)

Return true if the DEMs are required for later processing

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

get_result_file_name(key, index)

Return the name of the file to save.

Parameters:

• key (str)

• index (int)

Return type:

str

get_result_file_path(key, index=None)

Return the file name of the file to save with the local cache path.

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

• index (int)

Return type:

pathlib.Path

waterway_elevations_exists()

Check to see if the waterway and culvert bathymeties have already been estimated.

minimum_elevation_in_polygon(geometry, dem)

Determine the minimum value in each polygon. Select only coordinates within the polygon bounding box before clipping to the bounding box and then returning the minimum elevation.

Parameters:

• geometry (shapely.geometry.Polygon)

• dem (xarray.Dataset)

estimate_closed_elevations(waterways)

Sample the DEM around the tunnels to estimate the bed elevation.

Parameters:

waterways (geopandas.GeoDataFrame)

estimate_open_elevations(waterways)

Sample the DEM along the open waterways to enforce a decreasing elevation.

Parameters:

waterways (geopandas.GeoDataFrame)

create_dem(waterways)

Create and return a DEM at a resolution 1.5x the waterway width.

Parameters:

waterways (geopandas.GeoDataFrame)

Return type:

xarray.Dataset

load_waterways()

Download OpenStreetMap waterways and tunnels within the catchment BBox.

Return type:

bool

run()

This method runs a pipeline that: * downloads all tunnels and waterways within a catchment. * creates and samples a DEM around each feature to estimate the bed

elevation.

• saves out extents and bed elevations of the waterway and tunnel network

class geofabrics.processor.StopbankCrestElevationEstimator(json_instructions, debug=True)

Bases: BaseProcessor

StopbankCrestElevationEstimator executes a pipeline to estimate stopbank crest elevations. A DEM is generated of the surrounding area and this used to estimate crest elevations by taking the highest value in the area around the stopebank.

Parameters:

• json_instructions (json)

• debug (bool)

load_dem(filename)

Load a DEM

Parameters:

filename (pathlib.Path)

get_stopbanks_instruction(key)

Return true if the DEMs are required for later processing

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

get_result_file_name(key, index)

Return the name of the file to save.

Parameters:

• key (str)

• index (int)

Return type:

str

get_result_file_path(key, index=None)

Return the file name of the file to save with the local cache path.

Parameters:

• behaviour (instructions The json instructions defining the)

• key (str)

• index (int)

Return type:

pathlib.Path

stopbanks_elevations_exists()

Check to see if the waterway and culvert bathymeties have already been estimated.

maximum_elevation_in_polygon(geometry, dem)

Determine the minimum value in each polygon. Select only coordinates within the polygon bounding box before clipping to the bounding box and then returning the minimum elevation.

Parameters:

• geometry (shapely.geometry.Polygon)

• dem (xarray.Dataset)

estimate_elevations_simple(stopbanks)

Sample the DEM around the tunnels to estimate the bed elevation.

Parameters:

stopbanks (geopandas.GeoDataFrame)

create_dem(stopbanks)

Create and return a DEM at a resolution 1.5x the waterway width.

Parameters:

stopbanks (geopandas.GeoDataFrame)

Return type:

xarray.Dataset

load_stopbanks()

Download OpenStreetMap waterways and tunnels within the catchment BBox.

Return type:

bool

run()

This method runs a pipeline that: * downloads all tunnels and waterways within a catchment. * creates and samples a DEM around each feature to estimate the bed

elevation.

• saves out extents and bed elevations of the waterway and tunnel network