Input Formats
Input data required for the process. Each input type has to be assigned an
identifier, wich then can be referenced from the mp.open()
function
from within the process.
Single file inputs
Mapchete can guess the driver from single file paths. In general, all formats readable by
rasterio
and fiona
will be read as NumPy
arrays and GeoJSON-like feature
dictionaries respectively.
Alternatively, a .mapchete
file can be provided. In this case, the data type read
depends on the output driver specified in the .mapchete
file. It is well-intended that
mapchete processes can be chained in such a way!
For single files like GeoTIFFs, JPEG2000 files, Shapefiles or GeoJSON files, a file path (either remote or local) is sufficient.
Example:
input:
dem: path/to/elevation_data.tif
land_polygons: path/to/land_polygons.shp
Processing performance is best if the input files support quick access to subsets of the data. For rasters, converting them to COG (Cloud Optimized GeoTIFF) can sometimes do wonders especially if the data sits on an object storage. For vector data a format which support spatial indexing (e.g. GeoPackage or FlatGeoBuf) can do similar tricks although this works just in a limited way if the data is on an object storage.
Example:
input:
dem:
format: raster_file
path: path/to/elevation_data.tif
cache:
path: tmp/path/to/elevation_data.tif
format: COG
land_polygons:
format: vector_file
path: path/to/land_polygons.shp
cache:
path: tmp/path/to/land_polygons.fgb
format: FlatGeobuf
keep: false # if set to false cached data will not be deleted after the process finishes
In some cases it can be beneficial if the data is not cached on disk but directly in memory. This feature should be used with care and only for smaller datasets.
Example:
input:
dem:
format: raster_file
path: path/to/elevation_data.tif
cache: memory
land_polygons:
format: vector_file
path: path/to/land_polygons.shp
cache: memory
It is also possible to define input data groups e.g. for extracted Sentinel-2 granules, where bands are stored in separate files:
Example:
input:
sentinel2_granule:
red: path/to/B04.jp2
green: path/to/B03.jp2
blue: path/to/B02.jp2
In case the inputs are stored on separate storages with individual access settings, they can be provided in a storage_options mapping:
Example:
input:
foo:
format: raster_file
path: s3://bucket1/image.tif
storage_options:
AWS_ACCESS_KEY_ID: some_key_id
AWS_SECRET_ACCESS_KEY: some_key_secret
foo2:
format: raster_file
path: s3://bucket2/image.tif
storage_options:
AWS_ACCESS_KEY_ID: some_other_key_id
AWS_SECRET_ACCESS_KEY: some_other_key_secret
It is not recommended to put in access credatials as plain text into the configuration. It is also possible to point to environment variables instead of values:
Example:
input:
foo:
format: raster_file
path: s3://bucket1/image.tif
storage_options:
AWS_ACCESS_KEY_ID: ${SOME_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${SOME_KEY_SECRET}
foo2:
format: raster_file
path: s3://bucket2/image.tif
storage_options:
AWS_ACCESS_KEY_ID: ${SOME_OTHER_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${SOME_OTHER_KEY_SECRET}
TileDirectory inputs
It is also possible to directly point to a TileDirectory
output path from another
mapchete process. This is very similar to provide a .mapchete
file path but with the
convenience to just refer to the path.
Example:
input:
foo: path_to_tiledirectory
Sometimes it can be beneficial to pass on some default values to a TileDirectory, such as the maximum zoom level available. In that case Mapchete knows to read data from this zoom level in case a process runs on a higher zoom.
Example:
input:
foo:
format: TileDirectory
path: path_to_tiledirectory
resampling: bilinear
max_zoom: 8 # now data can be read also from e.g. zoom 9 and will be resampled
Additional output formats
Additional drivers can be written and installed. They have to be registered to the
mapchete.formats.drivers
entrypoint from within the driver’s setup.py
file.