geofabrics.processor ==================== .. py:module:: geofabrics.processor .. autoapi-nested-parse:: This module contains classes associated with generating GeoFabric layers from LiDAR and bathymetry contours based on the instructions contained in a JSON file. GeoFabric layers include hydrologically conditioned DEMs. Classes ------- .. autoapisummary:: geofabrics.processor.BaseProcessor geofabrics.processor.RawLidarDemGenerator geofabrics.processor.HydrologicDemGenerator geofabrics.processor.PatchDemGenerator geofabrics.processor.RoughnessLengthGenerator geofabrics.processor.MeasuredRiverGenerator geofabrics.processor.RiverBathymetryGenerator geofabrics.processor.WaterwayBedElevationEstimator geofabrics.processor.StopbankCrestElevationEstimator Module Contents --------------- .. py:class:: BaseProcessor(json_instructions) Bases: :py:obj:`abc.ABC` An abstract class with general methods for accessing elements in instruction files including populating default values. Also contains functions for downloading remote data using geopais, and constructing data file lists. The `BaseProcessor` class contains several important class members: * instructions - Defines the pipeline execution instructions * catchment_geometry - Defines all relevant regions in a catchment required in the generation of a DEM as polygons. * logger - logging within this class .. py:method:: create_metadata() A clase to create metadata to be added as netCDF attributes. .. py:method:: get_instruction_path(key, defaults = {}) Return the file path from the instruction file, or default if there is a default value and the local cache is specified. Raise an error if the key is not in the instructions. .. py:method:: check_instruction_path(key) Return True if the file path exists in the instruction file, or True if there is a default value and the local cache is specified. .. py:method:: create_results_folder() Ensure the results folder has been created. .. py:method:: save_dem(filename, dataset, generator, compression) Save out the dem/geofabrics labelled array. :param filename: The filename including extension to save the DEM/geofabric to. :param dataset: The labelled array data to save. :param generator: The dem.DemBase object with a 'save_dem' function. .. py:method:: get_resolution() Return the resolution from the instruction file. Raise an error if not in the instructions. .. py:method:: get_crs() Return the CRS projection information (horiztonal and vertical) from the instruction file. Raise an error if 'output' is not in the instructions. If no 'crs' or 'horizontal' or 'vertical' values are specified then use the default value for each one missing from the instructions. If the default is used it is added to the instructions. .. py:method:: get_instruction_general(key, subkey = None) Return the general instruction from the instruction file or return the default value if not specified in the instruction file. Raise an error if the key is not in the instructions and there is no default value. If the default is used it is added to the instructions. .. py:method:: get_processing_instructions(key) Return the processing instruction from the instruction file or return the default value if not specified in the instruction file. If the default is used it is added to the instructions. :param key: The string identifying the instruction .. py:method:: check_datasets(key, data_type) Check to see if the dataset is included in the instructions: key = dataservice (i.e. local, opentogaphy, linxz, lris) :param key: The string identifying how the data is accessed (e.g. local, opentopography, linz, etc) :param data_type: The string identifying the data type (e.g. raster, lidar, vector) .. py:method:: check_vector_or_raster(key, api_type) Check to see if vector or raster key (i.e. land, ocean_contours, etc) is included either as a file path, or within any of API's (i.e. LINZ or LRIS). :param key: The string identifying the vector/raster :param api_type: The string identifying if the key is a vector or raster .. py:method:: get_vector_or_raster_paths(key, data_type, required = True) Get the path to the vector/raster key data included either as a file path or as an API. Return all paths where the vector key is specified. In the case that an API is specified ensure the data is fetched as well. :param key: The string identifying the vector/raster :param data_type: The string identifying if the key is a vector or raster :param required: If `True`, an exception will be raised if no path(s) are found. .. py:method:: get_lidar_dataset_crs(data_service, dataset_name) Checks to see if source CRS of an associated LiDAR dataset has be specified in the instruction file. If it has been specified, this CRS is returned, and will later be used to override the CRS encoded in the LAS files. .. py:method:: get_lidar_datasets_info() Return a dictionary with three enties 'file_paths', 'crs' and 'tile_index_file'. The 'file_paths' contains a list of LiDAR tiles to process. The 'crs' (or coordinate system of the LiDAR data as defined by an EPSG code) is only optionally set (if unset the value is None). The 'crs' should only be set if the CRS information is not correctly encoded in this is only supported for OpenTopography LiDAR. The 'tile_index_file' is also optional (if unset the value is None). The 'tile_index_file' should be given if a tile index file exists for the LiDAR files specifying the extents of each tile. This is currently only supported for OpenTopography files. If a LiDAR dataset (either through an API or locally) is specified this is checked and all files within the catchment area are downloaded and used to construct the file list. If none is specified, the instruction 'data_paths' is checked for 'lidars' and these are returned. .. py:method:: run() :abstractmethod: This method controls the processor execution and code-flow. .. py:class:: RawLidarDemGenerator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` RawLidarDemGenerator executes a pipeline for creating a DEM from LiDAR and optionally a coarse DEM. The data sources and pipeline logic is defined in the json_instructions file. See the GitHub repository wiki or GeoFabrics/tests/ for usage examples and other documentation. .. py:method:: run() This method executes the geofabrics generation pipeline to produce geofabric derivatives. Note it currently only considers one LiDAR dataset that can have many tiles. See 'get_lidar_datasets_info' for where to change this. .. py:class:: HydrologicDemGenerator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` HydrologicDemGenerator executes a pipeline for loading in a DEM then incorporating hydrological features (e.g. ocean bathymetry, rivers, waterways,stopbank creast heights) to produce a hydrologically conditioned DEM. The data and pipeline logic is defined in the json_instructions file. See the GitHub repository wiki or GeoFabrics/tests/ for usage examples and other documentation. .. py:method:: add_hydrological_features(hydrologic_dem, area_threshold, catchment_dirs, temp_folder) Add in any bathymetry data - ocean or river .. py:method:: run() This method executes the geofabrics generation pipeline to produce geofabric derivatives. .. py:class:: PatchDemGenerator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` PatchDemGenerator executes a pipeline for loading in a DEM / roughness GeoFabrics profuct before adding specified patches to it. See the README.md for usage examples or GeoFabrics/tests/ for examples of usage and an instruction file .. py:method:: get_patch_instruction(key) Return true if the DEMs are required for later processing :param instructions The json instructions defining the behaviour: .. py:method:: run() This method executes the geofabrics generation pipeline to produce geofabric derivatives. .. py:class:: RoughnessLengthGenerator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` RoughnessLengthGenerator executes a pipeline for loading in a hydrologically conditioned DEM and LiDAR tiles to produce a roughness length layer that is added to the Hydrologically conditioned DEM. The data and pipeline logic is defined in the json_instructions file. The `RoughnessLengthGenerator` class contains several important class members: * catchment_geometry - Defines all relevant regions in a catchment required in the generation of a DEM as polygons. * roughness_dem - Adds a roughness layer to the hydrologically conditioned DEM. .. py:method:: get_roughness_instruction(key) Return true if the DEMs are required for later processing :param instructions The json instructions defining the behaviour: .. py:method:: load_roads_osm() Download OpenStreetMap roads within the catchment BBox. .. py:method:: run() This method executes the geofabrics generation pipeline to produce geofabric derivatives. .. py:class:: MeasuredRiverGenerator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` MeasuredRiverGenerator executes a pipeline to interpolate between measured river cross section elevations. A json_instructions file defines the pipeline logic and data. .. py:method:: get_measured_instruction(key) Return true if the DEMs are required for later processing :param instructions The json instructions defining the behaviour: .. py:method:: estimate_river_mouth_fan(defaults) Calculate and save depth estimates along the river mouth fan. .. py:method:: run() This method extracts a main channel then executes the DemGeneration pipeline to produce a DEM before sampling this to extimate width, slope and eventually depth. .. py:class:: RiverBathymetryGenerator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` RiverbathymetryGenerator executes a pipeline to estimate river bathymetry depths from flows, slopes, friction and widths along a main channel. This is dones by first creating a hydrologically conditioned DEM of the channel. A json_instructions file defines the pipeline logic and data. .. attribute:: channel_polyline The main channel along which to estimate depth. This is a polyline. .. attribute:: gen_dem The ground DEM generated along the main channel. This is a raster. .. attribute:: veg_dem The vegetation DEM generated along the main channel. This is a raster. .. attribute:: aligned_channel_polyline The main channel after its alignment has been updated based on the DEM. Width and slope are estimated on this. .. attribute:: transects Transect polylines perpindicular to the aligned channel with samples of the DEM values .. py:method:: channel_characteristics_exist() Return true if the DEMs are required for later processing .. py:method:: channel_bathymetry_exist() Return true if the river channel and bathymetry files exist. .. py:method:: alignment_exists() Return true if the DEMs are required for later processing :param instructions The json instructions defining the behaviour: .. py:method:: get_result_file_name(key = None, name = None) Return the file name of the file to save. :param instructions The json instructions defining the behaviour: .. py:method:: get_result_file_path(key = None, name = None) Return the file name of the file to save with the local cache path. :param instructions The json instructions defining the behaviour: .. py:method:: get_bathymetry_instruction(key) Return true if the DEMs are required for later processing :param instructions The json instructions defining the behaviour: .. py:method:: get_network_channel() Read in or create a channel from a river network. .. py:method:: get_dems(channel) Allow selection of the ground or vegetation DEM, and either create or load it. :param instructions The json instructions defining the behaviour: .. py:method:: align_channel(channel_width, channel) Align the river network defined channel based on LiDAR and save the aligned channel. :param channel_width The class for characterising channel width and other: properties :param channel The river network defined channel alignment: .. py:method:: calculate_channel_characteristics(channel_width, aligned_channel) Align the river network defined channel based on LiDAR and save the aligned channel. :param channel_width The class for characterising channel width and other: properties :param channel The river network defined channel alignment: .. py:method:: characterise_channel() Calculate the channel width, slope and other characteristics. This requires a ground and vegetation DEM. This also may require alignment of the channel centreline. .. py:method:: align_channel_from_rec() Calculate the channel width, slope and other characteristics. This requires a ground and vegetation DEM. This also may require alignment of the channel centreline. .. py:method:: align_channel_from_osm() Calculate the channel width, slope and other characteristics. This requires a ground and vegetation DEM. This also may require alignment of the channel centreline. .. py:method:: calculate_river_bed_elevations() Calculate and save depth estimates along the channel using various approaches. .. py:method:: estimate_river_mouth_fan() Calculate and save depth estimates along the river mouth fan. .. py:method:: run() This method extracts a main channel then executes the DemGeneration pipeline to produce a DEM before sampling this to extimate width, slope and eventually depth. .. py:class:: WaterwayBedElevationEstimator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` WaterwayBedElevationGenerator executes a pipeline to pull in OpenStreetMap waterway and tunnel information. A DEM is generated of the surrounding area and this used to unblock waterways and tunnels - by taking the lowest value in the area around a tunnel to be the tunnel elevation, and ensuring opne waterways flow downhill. .. py:method:: load_dem(filename) Load a DEM .. py:method:: get_waterways_instruction(key) Return true if the DEMs are required for later processing :param instructions The json instructions defining the behaviour: .. py:method:: get_result_file_name(key, index) Return the name of the file to save. .. py:method:: get_result_file_path(key, index = None) Return the file name of the file to save with the local cache path. :param instructions The json instructions defining the behaviour: .. py:method:: waterway_elevations_exists() Check to see if the waterway and culvert bathymeties have already been estimated. .. py:method:: minimum_elevation_in_polygon(geometry, dem) Determine the minimum value in each polygon. Select only coordinates within the polygon bounding box before clipping to the bounding box and then returning the minimum elevation. .. py:method:: estimate_closed_elevations(waterways) Sample the DEM around the tunnels to estimate the bed elevation. .. py:method:: estimate_open_elevations(waterways) Sample the DEM along the open waterways to enforce a decreasing elevation. .. py:method:: create_dem(waterways) Create and return a DEM at a resolution 1.5x the waterway width. .. py:method:: load_waterways() Download OpenStreetMap waterways and tunnels within the catchment BBox. .. py:method:: run() This method runs a pipeline that: * downloads all tunnels and waterways within a catchment. * creates and samples a DEM around each feature to estimate the bed elevation. * saves out extents and bed elevations of the waterway and tunnel network .. py:class:: StopbankCrestElevationEstimator(json_instructions, debug = True) Bases: :py:obj:`BaseProcessor` StopbankCrestElevationEstimator executes a pipeline to estimate stopbank crest elevations. A DEM is generated of the surrounding area and this used to estimate crest elevations by taking the highest value in the area around the stopebank. .. py:method:: load_dem(filename) Load a DEM .. py:method:: get_stopbanks_instruction(key) Return true if the DEMs are required for later processing :param instructions The json instructions defining the behaviour: .. py:method:: get_result_file_name(key, index) Return the name of the file to save. .. py:method:: get_result_file_path(key, index = None) Return the file name of the file to save with the local cache path. :param instructions The json instructions defining the behaviour: .. py:method:: stopbanks_elevations_exists() Check to see if the waterway and culvert bathymeties have already been estimated. .. py:method:: maximum_elevation_in_polygon(geometry, dem) Determine the minimum value in each polygon. Select only coordinates within the polygon bounding box before clipping to the bounding box and then returning the minimum elevation. .. py:method:: estimate_elevations_simple(stopbanks) Sample the DEM around the tunnels to estimate the bed elevation. .. py:method:: create_dem(stopbanks) Create and return a DEM at a resolution 1.5x the waterway width. .. py:method:: load_stopbanks() Download OpenStreetMap waterways and tunnels within the catchment BBox. .. py:method:: run() This method runs a pipeline that: * downloads all tunnels and waterways within a catchment. * creates and samples a DEM around each feature to estimate the bed elevation. * saves out extents and bed elevations of the waterway and tunnel network