mapchete package

Subpackages

Submodules

Module contents

class mapchete.Mapchete(config, with_cache=False)

Bases: object

Main entry point to every processing job.

From here, the process tiles can be determined and executed.

Parameters
configMapcheteConfig

Mapchete process configuration

with_cachebool

cache processed output data in memory (default: False)

Attributes
configMapcheteConfig

Mapchete process configuration

with_cachebool

process output data cached in memory

batch_preprocess(dask_scheduler=None, dask_max_submitted_tasks=500, workers=None, executor=None)

Run all required preprocessing steps.

Parameters
dask_schedulterstr

URL to a dask scheduler if distributed execution is desired

dask_max_submitted_tasksint

Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)

workersint

number of workers to be used for local processing

executormapchete.Executor

optional executor class to be used for processing

batch_preprocessor(dask_scheduler=None, dask_max_submitted_tasks=500, workers=None, executor=None)

Run all required preprocessing steps and yield over results.

The task count can be determined by self.config.preprocessing_tasks_count().

Parameters
dask_schedulterstr

URL to a dask scheduler if distributed execution is desired

dask_max_submitted_tasksint

Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)

workersint

number of workers to be used for local processing

executormapchete.Executor

optional executor class to be used for processing

batch_process(zoom=None, tile=None, dask_scheduler=None, dask_max_submitted_tasks=500, multi=None, workers=None, multiprocessing_module=None, multiprocessing_start_method='spawn', skip_output_check=False, executor=None)

Process a large batch of tiles.

Parameters
processMapcheteProcess

process to be run

zoomlist or int

either single zoom level or list of minimum and maximum zoom level; None processes all (default: None)

tiletuple

zoom, row and column of tile to be processed (cannot be used with zoom)

workersint

number of workers (default: number of CPU cores)

dask_schedulterstr

URL to a dask scheduler if distributed execution is desired

dask_max_submitted_tasksint

Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)

multiprocessing_modulemodule

either Python’s standard ‘multiprocessing’ or Celery’s ‘billiard’ module (default: multiprocessing)

multiprocessing_start_methodstr

“fork”, “forkserver” or “spawn” (default: “spawn”)

skip_output_checkbool

skip checking whether process tiles already have existing output before starting to process;

executormapchete.Executor

optional executor class to be used for processing

batch_processor(zoom=None, tile=None, dask_scheduler=None, dask_max_submitted_tasks=500, multi=None, workers=None, multiprocessing_module=None, multiprocessing_start_method='spawn', skip_output_check=False, executor=None)

Process a large batch of tiles and yield report messages per tile.

Parameters
zoomlist or int

either single zoom level or list of minimum and maximum zoom level; None processes all (default: None)

tiletuple

zoom, row and column of tile to be processed (cannot be used with zoom)

dask_schedulterstr

URL to a dask scheduler if distributed execution is desired

dask_max_submitted_tasksint

Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)

multiint

number of workers (default: number of CPU cores)

multiprocessing_modulemodule

either Python’s standard ‘multiprocessing’ or Celery’s ‘billiard’ module (default: multiprocessing)

multiprocessing_start_methodstr

“fork”, “forkserver” or “spawn” (default: “spawn”)

skip_output_checkbool

skip checking whether process tiles already have existing output before starting to process;

executormapchete.Executor

optional executor class to be used for processing

count_tasks(minzoom=None, maxzoom=None, init_zoom=0)

Count all preprocessing tasks and tiles at given zoom levels.

Parameters
minzoomint

limits minimum process zoom

maxzoomint

limits maximum process zoom

init_zoomint

initial zoom level used for tile count algorithm

Returns
number of tasks
count_tiles(minzoom=None, maxzoom=None, init_zoom=0)

Count number of tiles intersecting with process area at given zoom levels.

Parameters
minzoomint

limits minimum process zoom

maxzoomint

limits maximum process zoom

init_zoomint

initial zoom level used for tile count algorithm

Returns
number of tiles
execute(process_tile, raise_nodata=False)

Run Mapchete process on a tile.

Execute, write and return data.

Parameters
process_tileTile or tile index tuple

Member of the process tile pyramid (not necessarily the output pyramid, if output has a different metatiling setting)

Returns
dataNumPy array or features

process output

get_process_tiles(zoom=None, batch_by=None)

Yield process tiles.

Tiles intersecting with the input data bounding boxes as well as process bounds, if provided, are considered process tiles. This is to avoid iterating through empty tiles.

Parameters
zoominteger

zoom level process tiles should be returned from; if none is given, return all process tiles

batch_bystr

if not None, tiles can be yielded in batches either by “row” or “column”.

Yields
BufferedTile objects
get_raw_output(tile, _baselevel_readonly=False)

Get output raw data.

This function won’t work with multiprocessing, as it uses the threading.Lock() class.

Parameters
tiletuple, Tile or BufferedTile

If a tile index is given, a tile from the output pyramid will be assumed. Tile cannot be bigger than process tile!

Returns
dataNumPy array or features

process output

read(output_tile)

Read from written process output.

Parameters
output_tileBufferedTile or tile index tuple

Member of the output tile pyramid (not necessarily the process pyramid, if output has a different metatiling setting)

Returns
dataNumPy array or features

process output

skip_tiles(tiles=None, tiles_batches=None)

Quickly determine whether tiles can be skipped for processing.

The skip value is True if process mode is ‘continue’ and process output already exists. In all other cases, skip is False.

Parameters
tileslist of process tiles
Yields
tuples(tile, skip)
write(process_tile, data)

Write data into output format.

Parameters
process_tileBufferedTile or tile index tuple

process tile

dataNumPy array or features

data to be written

class mapchete.MapcheteProcess(tile=None, params=None, input=None, output_params=None, config=None)

Bases: object

Process class inherited by user process script.

Its attributes and methods can be accessed via “self” from within a Mapchete process Python file.

Parameters
tileBufferedTile

Tile process should be run on

configMapcheteConfig

process configuration

paramsdictionary

process parameters

Attributes
identifierstring

process identifier

titlestring

process title

versionstring

process version string

abstractstring

short text describing process purpose

tileBufferedTile

Tile process should be run on

tile_pyramidBufferedTilePyramid

process tile pyramid

output_pyramidBufferedTilePyramid

output tile pyramid

paramsdictionary

process parameters

clip(array, geometries, inverted=False, clip_buffer=0)

Clip array by geometry.

Parameters
arrayarray

raster data to be clipped

geometriesiterable

geometries used to clip source array

invertedbool

invert clipping (default: False)

clip_bufferint

buffer (in pixels) geometries before applying clip

Returns
clipped arrayarray
contours(elevation, interval=100, field='elev', base=0)

Extract contour lines from elevation data.

Parameters
elevationarray

input elevation data

intervalinteger

elevation value interval when drawing contour lines

fieldstring

output field name containing elevation value

baseinteger

elevation base value the intervals are computed from

Returns
contoursiterable

contours as GeoJSON-like pairs of properties and geometry

hillshade(elevation, azimuth=315.0, altitude=45.0, z=1.0, scale=1.0)

Calculate hillshading from elevation data.

Parameters
elevationarray

input elevation data

azimuthfloat

horizontal angle of light source (315: North-West)

altitudefloat

vertical angle of light source (90 would result in slope shading)

zfloat

vertical exaggeration factor

scalefloat

scale factor of pixel size units versus height units (insert 112000 when having elevation values in meters in a geodetic projection)

Returns
hillshadearray
open(**kwargs)
read(**kwargs)

Read existing output data from a previous run.

Returns
process outputNumPy array (raster) or feature iterator (vector)
write(data, **kwargs)

Deprecated.

class mapchete.ProcessInfo(tile, processed, process_msg, written, write_msg)

Bases: tuple

property process_msg

Alias for field number 2

property processed

Alias for field number 1

property tile

Alias for field number 0

property write_msg

Alias for field number 4

property written

Alias for field number 3

class mapchete.Timer(elapsed=0.0, str_round=3)

Bases: object

Context manager to facilitate timing code.

Examples

>>> with Timer() as t:
        ...  # some longer running code
>>> print(t)  # prints elapsed time

based on http://preshing.com/20110924/timing-your-code-using-pythons-with-statement/

property elapsed
mapchete.count_tiles(geometry, pyramid, minzoom, maxzoom, init_zoom=0, rasterize_threshold=0)

Count number of tiles intersecting with geometry.

Parameters
geometryshapely geometry
pyramidTilePyramid
minzoomint
maxzoomint
init_zoomint
Returns
number of tiles
mapchete.open(some_input, with_cache=False, fs=None, fs_kwargs=None, **kwargs)

Open a Mapchete process.

Parameters
some_inputMapcheteConfig object, config dict, path to mapchete file or path to

TileDirectory Mapchete process configuration

modestring
  • memory: Generate process output on demand without reading pre-existing data or writing new data.

  • readonly: Just read data without processing new data.

  • continue: (default) Don’t overwrite existing output.

  • overwrite: Overwrite existing output.

zoomlist or integer

process zoom level or a pair of minimum and maximum zoom level

boundstuple

left, bottom, right, top process boundaries in output pyramid

single_input_filestring

single input file if supported by process

with_cachebool

process output data cached in memory

fsfsspec FileSystem

Any FileSystem object for the mapchete output.

fs_kwargsdict

Special configuration parameters if FileSystem object has to be created.

Returns
Mapchete

a Mapchete process object