mapchete package¶

Subpackages¶

Submodules¶

Module contents¶

class mapchete.Executor(*args, concurrency=None, **kwargs)¶

Bases: object

Executor factory for dask and concurrent.futures executor

Will move into the mapchete core package.

class mapchete.FakeFuture(func, fargs=None, fkwargs=None)¶

Bases: object

Wrapper class to mimick future interface.

cancelled()¶: Sequential futures cannot be cancelled.

exception(**kwargs)¶: Raise task exception if any.

result(**kwargs)¶: Return task result.

class mapchete.Job(func: Generator, fargs: Optional[tuple] = None, fkwargs: Optional[dict] = None, as_iterator: bool = False, tiles_tasks: Optional[int] = None, preprocessing_tasks: Optional[int] = None, executor_concurrency: str = 'processes', executor_kwargs: Optional[dict] = None, process_area=None, stac_item_path: Optional[str] = None)¶

Bases: object

Wraps the output of a processing function into a generator with known length.

This class also exposes the internal Executor.cancel() function in order to cancel all remaining tasks/futures.

Will move into the mapchete core package.

cancel()¶: Cancel all running and pending Job tasks.

set_executor_concurrency(executor_concurrency)¶

Overwrite default or previously set Executor concurrency.

This only has an effect if Job was initialized with ‘as_iterator’ and has not yet run.

set_executor_kwargs(executor_kwargs)¶

Overwrite default or previously set Executor creation kwargs.

This only has an effect if Job was initialized with ‘as_iterator’ and has not yet run.

class mapchete.Mapchete(config, with_cache=False)¶

Bases: object

Main entry point to every processing job.

From here, the process tiles can be determined and executed.

Parameters

configMapcheteConfig: Mapchete process configuration
with_cachebool: cache processed output data in memory (default: False)

Attributes

configMapcheteConfig: Mapchete process configuration
with_cachebool: process output data cached in memory

batch_preprocess(dask_scheduler=None, dask_max_submitted_tasks=500, dask_chunksize=100, workers=None, executor=None)¶

Run all required preprocessing steps.

Parameters

dask_schedulterstr: URL to a dask scheduler if distributed execution is desired
dask_max_submitted_tasksint: Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)
dask_chunksizeint: Number of tasks submitted to the scheduler at once. (default: 100)
workersint: number of workers to be used for local processing
executormapchete.Executor: optional executor class to be used for processing

batch_preprocessor(dask_scheduler=None, dask_max_submitted_tasks=500, dask_chunksize=100, workers=None, executor=None)¶

Run all required preprocessing steps and yield over results.

The task count can be determined by self.config.preprocessing_tasks_count().

Parameters

dask_schedulterstr: URL to a dask scheduler if distributed execution is desired
dask_max_submitted_tasksint: Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)
dask_chunksizeint: Number of tasks submitted to the scheduler at once. (default: 100)
workersint: number of workers to be used for local processing
executormapchete.Executor: optional executor class to be used for processing

batch_process(zoom=None, tile=None, dask_scheduler=None, dask_max_submitted_tasks=500, dask_chunksize=100, multi=None, workers=None, multiprocessing_module=None, multiprocessing_start_method='spawn', skip_output_check=False, executor=None)¶

Process a large batch of tiles.

Parameters

processMapcheteProcess: process to be run
zoomlist or int: either single zoom level or list of minimum and maximum zoom level; None processes all (default: None)
tiletuple: zoom, row and column of tile to be processed (cannot be used with zoom)
workersint: number of workers (default: number of CPU cores)
dask_schedulterstr: URL to a dask scheduler if distributed execution is desired
dask_max_submitted_tasksint: Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)
dask_chunksizeint: Number of tasks submitted to the scheduler at once. (default: 100)
multiprocessing_modulemodule: either Python’s standard ‘multiprocessing’ or Celery’s ‘billiard’ module (default: multiprocessing)
multiprocessing_start_methodstr: “fork”, “forkserver” or “spawn” (default: “spawn”)
skip_output_checkbool: skip checking whether process tiles already have existing output before starting to process;
executormapchete.Executor: optional executor class to be used for processing

batch_processor(zoom=None, tile=None, dask_scheduler=None, dask_max_submitted_tasks=500, dask_chunksize=100, multi=None, workers=None, multiprocessing_module=None, multiprocessing_start_method='spawn', skip_output_check=False, executor=None)¶

Process a large batch of tiles and yield report messages per tile.

Parameters

zoomlist or int: either single zoom level or list of minimum and maximum zoom level; None processes all (default: None)
tiletuple: zoom, row and column of tile to be processed (cannot be used with zoom)
dask_schedulterstr: URL to a dask scheduler if distributed execution is desired
dask_max_submitted_tasksint: Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)
dask_chunksizeint: Number of tasks submitted to the scheduler at once. (default: 100)
multiint: number of workers (default: number of CPU cores)
multiprocessing_modulemodule: either Python’s standard ‘multiprocessing’ or Celery’s ‘billiard’ module (default: multiprocessing)
multiprocessing_start_methodstr: “fork”, “forkserver” or “spawn” (default: “spawn”)
skip_output_checkbool: skip checking whether process tiles already have existing output before starting to process;
executormapchete.Executor: optional executor class to be used for processing

compute(**kwargs)¶: Compute preprocessing tasks and tile tasks in one go.

count_tasks(minzoom=None, maxzoom=None, init_zoom=0)¶

Count all preprocessing tasks and tiles at given zoom levels.

Parameters

minzoomint: limits minimum process zoom
maxzoomint: limits maximum process zoom
init_zoomint: initial zoom level used for tile count algorithm

Returns

number of tasks

count_tiles(minzoom=None, maxzoom=None, init_zoom=0)¶

Count number of tiles intersecting with process area at given zoom levels.

Parameters

minzoomint: limits minimum process zoom
maxzoomint: limits maximum process zoom
init_zoomint: initial zoom level used for tile count algorithm

Returns

number of tiles

execute(process_tile, raise_nodata=False)¶

Run Mapchete process on a tile.

Execute, write and return data.

Parameters

process_tileTile or tile index tuple: Member of the process tile pyramid (not necessarily the output pyramid, if output has a different metatiling setting)

Returns

dataNumPy array or features: process output

get_process_tiles(zoom=None, batch_by=None)¶

Yield process tiles.

Tiles intersecting with the input data bounding boxes as well as process bounds, if provided, are considered process tiles. This is to avoid iterating through empty tiles.

Parameters

zoominteger: zoom level process tiles should be returned from; if none is given, return all process tiles
batch_bystr: if not None, tiles can be yielded in batches either by “row” or “column”.

Yields

BufferedTile objects

get_raw_output(tile, _baselevel_readonly=False)¶

Get output raw data.

This function won’t work with multiprocessing, as it uses the threading.Lock() class.

Parameters

tiletuple, Tile or BufferedTile: If a tile index is given, a tile from the output pyramid will be assumed. Tile cannot be bigger than process tile!

Returns

dataNumPy array or features: process output

read(output_tile)¶

Read from written process output.

Parameters

output_tileBufferedTile or tile index tuple: Member of the output tile pyramid (not necessarily the process pyramid, if output has a different metatiling setting)

Returns

dataNumPy array or features: process output

skip_tiles(tiles=None, tiles_batches=None)¶

Quickly determine whether tiles can be skipped for processing.

The skip value is True if process mode is ‘continue’ and process output already exists. In all other cases, skip is False.

Parameters

tileslist of process tiles

Yields

tuples(tile, skip)

task_batches(zoom=None, tile=None, skip_output_check=False, **kwargs)¶: Generate task batches from preprocessing tasks and tile tasks.

write(process_tile, data)¶

Write data into output format.

Parameters

process_tileBufferedTile or tile index tuple: process tile
dataNumPy array or features: data to be written

write_stac(indent=4)¶

Create or update existing STAC JSON file.

On TileDirectory outputs, the tiled-assets are updated regarding zoom levels and bounds.

class mapchete.MapcheteProcess(tile=None, params=None, input=None, output_params=None, config=None)¶

Bases: object

Process class inherited by user process script.

Its attributes and methods can be accessed via “self” from within a Mapchete process Python file.

Parameters

tileBufferedTile: Tile process should be run on
configMapcheteConfig: process configuration
paramsdictionary: process parameters

Attributes

identifierstring: process identifier
titlestring: process title
versionstring: process version string
abstractstring: short text describing process purpose
tileBufferedTile: Tile process should be run on
tile_pyramidBufferedTilePyramid: process tile pyramid
output_pyramidBufferedTilePyramid: output tile pyramid
paramsdictionary: process parameters

clip(array, geometries, inverted=False, clip_buffer=0)¶

Clip array by geometry.

Parameters

arrayarray: raster data to be clipped
geometriesiterable: geometries used to clip source array
invertedbool: invert clipping (default: False)
clip_bufferint: buffer (in pixels) geometries before applying clip

Returns

clipped arrayarray

contours(elevation, interval=100, field='elev', base=0)¶

Extract contour lines from elevation data.

Parameters

elevationarray: input elevation data
intervalinteger: elevation value interval when drawing contour lines
fieldstring: output field name containing elevation value
baseinteger: elevation base value the intervals are computed from

Returns

contoursiterable: contours as GeoJSON-like pairs of properties and geometry

hillshade(elevation, azimuth=315.0, altitude=45.0, z=1.0, scale=1.0)¶

Calculate hillshading from elevation data.

Parameters

elevationarray: input elevation data
azimuthfloat: horizontal angle of light source (315: North-West)
altitudefloat: vertical angle of light source (90 would result in slope shading)
zfloat: vertical exaggeration factor
scalefloat: scale factor of pixel size units versus height units (insert 112000 when having elevation values in meters in a geodetic projection)

Returns

hillshadearray

open(**kwargs)¶

read(**kwargs)¶

Read existing output data from a previous run.

Returns

process outputNumPy array (raster) or feature iterator (vector)

write(data, **kwargs)¶: Deprecated.

class mapchete.ProcessInfo(tile, processed, process_msg, written, write_msg, data)¶

Bases: tuple

data¶: Alias for field number 5

process_msg¶: Alias for field number 2

processed¶: Alias for field number 1

tile¶: Alias for field number 0

write_msg¶: Alias for field number 4

written¶: Alias for field number 3

class mapchete.SkippedFuture(result=None, skip_info=None, **kwargs)¶

Bases: object

Wrapper class to mimick future interface for empty tasks.

cancelled()¶: Nothing to cancel here.

exception(**kwargs)¶: Nothing to raise here.

result(**kwargs)¶: Only return initial result value.

class mapchete.Timer(elapsed=0.0, str_round=3)¶

Bases: object

Context manager to facilitate timing code.

Examples

>>> with Timer() as t:
        ...  # some longer running code
>>> print(t)  # prints elapsed time

based on http://preshing.com/20110924/timing-your-code-using-pythons-with-statement/

property elapsed¶

mapchete.count_tiles(geometry, pyramid, minzoom, maxzoom, init_zoom=0, rasterize_threshold=0)¶

Count number of tiles intersecting with geometry.

Parameters

geometryshapely geometry
pyramidTilePyramid
minzoomint
maxzoomint
init_zoomint

Returns

number of tiles

mapchete.open(some_input, with_cache=False, fs=None, fs_kwargs=None, **kwargs)¶

Open a Mapchete process.

Parameters

some_inputMapcheteConfig object, config dict, path to mapchete file or path to

TileDirectory Mapchete process configuration

modestring

memory: Generate process output on demand without reading pre-existing data or writing new data.
readonly: Just read data without processing new data.
continue: (default) Don’t overwrite existing output.
overwrite: Overwrite existing output.

zoomlist or integer

process zoom level or a pair of minimum and maximum zoom level

boundstuple

left, bottom, right, top process boundaries in output pyramid

single_input_filestring

single input file if supported by process

with_cachebool

process output data cached in memory

fsfsspec FileSystem

Any FileSystem object for the mapchete output.

fs_kwargsdict

Special configuration parameters if FileSystem object has to be created.

Returns

Mapchete: a Mapchete process object