mapchete.commands package

Module contents

This package contains easy to access functions which otherwise would have to be called via the CLI. This should make the use from within other scripts, notebooks, etc. easier.

mapchete.commands.convert(tiledir: Union[str, dict], output: str, zoom: Optional[Union[int, List[int]]] = None, area: Optional[Union[shapely.geometry.base.BaseGeometry, str, dict]] = None, area_crs: Optional[Union[rasterio.crs.CRS, str]] = None, bounds: Optional[Tuple[float]] = None, bounds_crs: Optional[Union[rasterio.crs.CRS, str]] = None, point: Optional[Tuple[float, float]] = None, point_crs: Optional[Tuple[float, float]] = None, tile: Optional[Tuple[int, int, int]] = None, overwrite: bool = False, concurrency: str = 'processes', dask_scheduler: Optional[str] = None, dask_max_submitted_tasks: int = 500, dask_chunksize: int = 100, dask_client=None, dask_compute_graph=True, workers: Optional[int] = None, multi: Optional[int] = None, clip_geometry: Optional[str] = None, bidx: Optional[List[int]] = None, output_pyramid: Optional[str] = None, output_metatiling: Optional[int] = None, output_format: Optional[str] = None, output_dtype: Optional[str] = None, output_geometry_type: Optional[str] = None, creation_options: Optional[dict] = None, scale_ratio: Optional[float] = None, scale_offset: Optional[float] = None, resampling_method: str = 'nearest', overviews: bool = False, overviews_resampling_method: str = 'cubic_spline', cog: bool = False, msg_callback: Optional[Callable] = None, as_iterator: bool = False) → Job

Convert mapchete outputs or other geodata.

This is a wrapper around the mapchete.processes.convert process which helps generating tiled outputs for raster and vector data or single COGs from TileDirectory raster inputs.

It also supports clipping of the input by a vector dataset.

If only a subset of a TileDirectory is desired, please see the mapchete.commands.cp command.

Parameters

tiledirstr

Path to TileDirectory or mapchete config.

outputstr

Path to output.

zoominteger or list of integers

Single zoom, minimum and maximum zoom or a list of zoom levels.

areastr, dict, BaseGeometry

Geometry to override bounds or area provided in process configuration. Can be either a WKT string, a GeoJSON mapping, a shapely geometry or a path to a Fiona-readable file.

area_crsCRS or str

CRS of area (default: process CRS).

boundstuple

Override bounds or area provided in process configuration.

bounds_crsCRS or str

CRS of area (default: process CRS).

pointiterable

X and y coordinates of point whose corresponding process tile bounds will be used.

point_crsstr or CRS

CRS of point (defaults to process pyramid CRS).

tiletuple

Zoom, row and column of tile to be processed (cannot be used with zoom)

overwritebool

Overwrite existing output.

workersint

Number of execution workers when processing concurrently.

concurrencystr

Concurrency to be used. Could either be “processes”, “threads” or “dask”.

dask_schedulerstr

URL to dask scheduler if required.

dask_max_submitted_tasksint

Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)

dask_chunksizeint

Number of tasks submitted to the scheduler at once. (default: 100)

dask_clientdask.distributed.Client

Reusable Client instance if required. Otherwise a new client will be created.

clip_geometrystr

Path to Fiona-readable file by which output will be clipped.

bidxlist of integers

Band indexes to read from source.

output_pyramidstr

Output pyramid to write to. Can either be one of the standard pyramid grids or a JSON file holding the grid definition.

output_metatilingint

Output metatiling.

output_formatstr

Output format. Can be any raster or vector format available by mapchete.

output_dtypestr

Output data type (for raster output only).

output_geometry_type

Output geometry type (for vector output only).

creation_optionsdict

Output driver specific creation options.

scale_ratiofloat

Scaling factor (for raster output only).

scale_offsetfloat

Scaling offset (for raster output only).

resampling_methodstr

Resampling method used. (default: nearest).

overviewsbool

Generate overviews (single GTiff output only).

overviews_resampling_methodstr

Resampling method used for overviews. (default: cubic_spline)

cogbool

Write a valid COG. This will automatically generate verviews. (GTiff only)

msg_callbackCallable

Optional callback function for process messages.

as_iteratorbool

Returns as generator but with a __len__() property.

Returns

mapchete.Job instance either with already processed items or a generator with known length.

Examples

>>> convert("foo", "bar")

This will run the whole conversion process.

>>> for i in convert("foo", "bar", as_iterator=True):
>>>     print(i)

This will return a generator where through iteration, tiles are copied.

>>> list(tqdm.tqdm(convert("foo", "bar", as_iterator=True)))

Usage within a process bar.

mapchete.commands.cp(src_tiledir: str, dst_tiledir: str, zoom: Optional[Union[int, List[int]]] = None, area: Optional[Union[shapely.geometry.base.BaseGeometry, str, dict]] = None, area_crs: Optional[Union[rasterio.crs.CRS, str]] = None, bounds: Optional[Tuple[float]] = None, bounds_crs: Optional[Union[rasterio.crs.CRS, str]] = None, point: Optional[Tuple[float, float]] = None, point_crs: Optional[Tuple[float, float]] = None, overwrite: bool = False, workers: Optional[int] = None, multi: Optional[int] = None, concurrency: Optional[str] = None, dask_scheduler: Optional[str] = None, dask_client=None, src_fs_opts: Optional[dict] = None, dst_fs_opts: Optional[dict] = None, msg_callback: Optional[Callable] = None, as_iterator: bool = False) → Job

Copy TileDirectory from source to destination.

Parameters

src_tiledirstr

Source TileDirectory or mapchete file.

dst_tiledirstr

Destination TileDirectory.

zoominteger or list of integers

Single zoom, minimum and maximum zoom or a list of zoom levels.

areastr, dict, BaseGeometry

Geometry to override bounds or area provided in process configuration. Can be either a WKT string, a GeoJSON mapping, a shapely geometry or a path to a Fiona-readable file.

area_crsCRS or str

CRS of area (default: process CRS).

boundstuple

Override bounds or area provided in process configuration.

bounds_crsCRS or str

CRS of area (default: process CRS).

pointiterable

X and y coordinates of point whose corresponding output tile bounds will be used.

point_crsstr or CRS

CRS of point (defaults to process pyramid CRS).

overwritebool

Overwrite existing output.

workersint

Number of threads used to check whether tiles exist.

concurrencystr

Concurrency to be used. Could either be “processes”, “threads” or “dask”.

dask_schedulerstr

URL to dask scheduler if required.

dask_clientdask.distributed.Client

Reusable Client instance if required. Otherwise a new client will be created.

src_fs_optsdict

Configuration options for source fsspec filesystem.

dst_fs_optsdict

Configuration options for destination fsspec filesystem.

msg_callbackCallable

Optional callback function for process messages.

as_iteratorbool

Returns as generator but with a __len__() property.

Returns

mapchete.Job instance either with already processed items or a generator with known length.

Examples

>>> cp("foo", "bar", zoom=5)

This will run the whole copy process.

>>> for i in cp("foo", "bar", zoom=5, as_iterator=True):
>>>     print(i)

This will return a generator where through iteration, tiles are copied.

>>> list(tqdm.tqdm(cp("foo", "bar", zoom=5, as_iterator=True)))

Usage within a process bar.

mapchete.commands.execute(mapchete_config: Union[str, dict], zoom: Optional[Union[int, List[int]]] = None, area: Optional[Union[shapely.geometry.base.BaseGeometry, str, dict]] = None, area_crs: Optional[Union[rasterio.crs.CRS, str]] = None, bounds: Optional[Tuple[float]] = None, bounds_crs: Optional[Union[rasterio.crs.CRS, str]] = None, point: Optional[Tuple[float, float]] = None, point_crs: Optional[Tuple[float, float]] = None, tile: Optional[Tuple[int, int, int]] = None, overwrite: bool = False, mode: str = 'continue', concurrency: str = 'processes', workers: Optional[int] = None, multi: Optional[int] = None, multiprocessing_start_method: Optional[str] = None, dask_scheduler: Optional[str] = None, dask_max_submitted_tasks=1000, dask_chunksize=100, dask_client=None, dask_compute_graph=True, dask_propagate_results=True, msg_callback: Optional[Callable] = None, as_iterator: bool = False) → Job

Execute a Mapchete process.

Parameters

mapchete_configstr or dict

Mapchete configuration as file path or dictionary.

zoominteger or list of integers

Single zoom, minimum and maximum zoom or a list of zoom levels.

areastr, dict, BaseGeometry

Geometry to override bounds or area provided in process configuration. Can be either a WKT string, a GeoJSON mapping, a shapely geometry or a path to a Fiona-readable file.

area_crsCRS or str

CRS of area (default: process CRS).

boundstuple

Override bounds or area provided in process configuration.

bounds_crsCRS or str

CRS of area (default: process CRS).

pointiterable

X and y coordinates of point whose corresponding process tile bounds will be used.

point_crsstr or CRS

CRS of point (defaults to process pyramid CRS).

tiletuple

Zoom, row and column of tile to be processed (cannot be used with zoom)

overwritebool

Overwrite existing output.

modestr

Set process mode. One of “readonly”, “continue” or “overwrite”.

workersint

Number of execution workers when processing concurrently.

multiprocessing_start_methodstr

Method used by multiprocessing module to start child workers. Availability of methods depends on OS.

concurrencystr

Concurrency to be used. Could either be “processes”, “threads” or “dask”.

dask_schedulerstr

URL to dask scheduler if required.

dask_max_submitted_tasksint

Make sure that not more tasks are submitted to dask scheduler at once. (default: 500)

dask_chunksizeint

Number of tasks submitted to the scheduler at once. (default: 100)

dask_clientdask.distributed.Client

Reusable Client instance if required. Otherwise a new client will be created.

dask_compute_graphbool

Build and compute dask graph instead of submitting tasks as preprocessing & zoom tiles batches. (default: True)

dask_propagate_resultsbool

Propagate results between tasks. This helps to minimize read calls when building overviews but can lead to a much higher memory consumption on the cluster. Only with effect if dask_compute_graph is activated. (default: True)

msg_callbackCallable

Optional callback function for process messages.

as_iteratorbool

Returns as generator but with a __len__() property.

Returns

mapchete.Job instance either with already processed items or a generator with known length.

Examples

>>> execute("foo")

This will run the whole execute process.

>>> for i in execute("foo", as_iterator=True):
>>>     print(i)

This will return a generator where through iteration, tiles are copied.

>>> list(tqdm.tqdm(execute("foo", as_iterator=True)))

Usage within a process bar.

mapchete.commands.index(tiledir: str, idx_out_dir: Optional[str] = None, geojson: bool = False, gpkg: bool = False, shp: bool = False, vrt: bool = False, txt: bool = False, fieldname: str = 'location', basepath: Optional[str] = None, for_gdal: bool = False, zoom: Optional[Union[int, List[int]]] = None, area: Optional[Union[shapely.geometry.base.BaseGeometry, str, dict]] = None, area_crs: Optional[Union[rasterio.crs.CRS, str]] = None, bounds: Optional[Tuple[float]] = None, bounds_crs: Optional[Union[rasterio.crs.CRS, str]] = None, point: Optional[Tuple[float, float]] = None, point_crs: Optional[Tuple[float, float]] = None, tile: Optional[Tuple[int, int, int]] = None, fs_opts: Optional[dict] = None, msg_callback: Optional[Callable] = None, as_iterator: bool = False, **_) → Job

Create one or more indexes from a TileDirectory.

Parameters

tiledirstr

Source TileDirectory or mapchete file.

idx_out_dirstr

Alternative output dir for index. Defaults to TileDirectory path.

geojsonbool

Activate GeoJSON output.

gpkgbool

Activate GeoPackage output.

shpbool

Activate Shapefile output.

vrtbool

Activate VRT output.

txtbool

Activate TXT output.

fieldnamestr

Field name which contains paths of tiles (default: “location”).

basepathstr

Use custom base path for absolute paths instead of output path.

for_gdalbool

Use GDAL compatible remote paths, i.e. add “/vsicurl/” before path.

zoominteger or list of integers

Single zoom, minimum and maximum zoom or a list of zoom levels.

areastr, dict, BaseGeometry

Geometry to override bounds or area provided in process configuration. Can be either a WKT string, a GeoJSON mapping, a shapely geometry or a path to a Fiona-readable file.

area_crsCRS or str

CRS of area (default: process CRS).

boundstuple

Override bounds or area provided in process configuration.

bounds_crsCRS or str

CRS of area (default: process CRS).

pointiterable

X and y coordinates of point whose corresponding process tile bounds will be used.

point_crsstr or CRS

CRS of point (defaults to process pyramid CRS).

tiletuple

Zoom, row and column of tile to be processed (cannot be used with zoom)

fs_optsdict

Configuration options for fsspec filesystem.

msg_callbackCallable

Optional callback function for process messages.

as_iteratorbool

Returns as generator but with a __len__() property.

Returns

mapchete.Job instance either with already processed items or a generator with known length.

Examples

>>> index("foo", vrt=True, zoom=5)

This will run the whole index process.

>>> for i in index("foo", vrt=True, zoom=5, as_iterator=True):
>>>     print(i)

This will return a generator where through iteration, tiles are copied.

>>> list(tqdm.tqdm(index("foo", vrt=True, zoom=5, as_iterator=True)))

Usage within a process bar.

mapchete.commands.rm(tiledir: str, zoom: Optional[Union[int, List[int]]] = None, area: Optional[Union[shapely.geometry.base.BaseGeometry, str, dict]] = None, area_crs: Optional[Union[rasterio.crs.CRS, str]] = None, bounds: Optional[Tuple[float]] = None, bounds_crs: Optional[Union[rasterio.crs.CRS, str]] = None, multi: Optional[int] = None, fs_opts: Optional[dict] = None, msg_callback: Optional[Callable] = None, as_iterator: bool = False) → Job

Remove tiles from TileDirectory.

Parameters

tiledirstr

TileDirectory or mapchete file.

zoominteger or list of integers

Single zoom, minimum and maximum zoom or a list of zoom levels.

areastr, dict, BaseGeometry

Geometry to override bounds or area provided in process configuration. Can be either a WKT string, a GeoJSON mapping, a shapely geometry or a path to a Fiona-readable file.

area_crsCRS or str

CRS of area (default: process CRS).

boundstuple

Override bounds or area provided in process configuration.

bounds_crsCRS or str

CRS of area (default: process CRS).

multiint

Number of threads used to check whether tiles exist.

fs_optsdict

Configuration options for fsspec filesystem.

msg_callbackCallable

Optional callback function for process messages.

as_iteratorbool

Returns as generator but with a __len__() property.

Returns

mapchete.Job instance either with already processed items or a generator with known length.

Examples

>>> rm("foo", zoom=5)

This will run the whole rm process.

>>> for i in rm("foo", zoom=5, as_iterator=True):
>>>     print(i)

This will return a generator where through iteration, tiles are removed.

>>> list(tqdm.tqdm(rm("foo", zoom=5, as_iterator=True)))

Usage within a process bar.