zetta_utils.layer

Description

zetta_utils.layer is an abstraction that allows reading from and writing to external data sources. Users interact with external data sources by creating data layers (zetta_utils.layer.Layer). A layer represents an indexable set of data, and is defined by an indexing scheme, IO backend, and a set of processors applied to the data/indexes during IO operations.

Indexing scheme defines what information will be passed to the IO backend and specifies how to translate user index inputs to a standardized format. IO backend defines how to retrieve the data given an index. IO backends may be used to support various data formats (eg CloudVolume, h5, DynamoDB), but also to achieve more interesting behaviours, such as grouping layers together, mapping index ranges to different data sources, etc.

Shortcuts for commonly used layer setups can be created For example, zetta_utils.layer.volumetric.cloudvol.build_cv_layer(...) will build a volumetric-indexed layer with a CloudVolume backend that includes processors that allow the user to index the layer at arbitrary resolutions.

Available Layer Setups

zetta_utils.layer.build_layer_set(layers, readonly=False, index_procs=(), read_procs=(), write_procs=())[source]

Build a generic layer set with mix/untyped set members. :type layers: Mapping[str, Layer] :param layers: Mapping from layer names to layers. :type readonly: bool :param readonly: Whether layer is read only. :type index_procs: Iterable[IndexProcessor] :param index_procs: List of processors that will be applied to the index given by the user

prior to IO operations.

Parameters

  • read_procs (Iterable[Union[DataProcessor[dict[str, ~DataT]], JointIndexDataProcessor[dict[str, ~DataT], ~IndexT]]]) – List of processors that will be applied to the read data before returning it to the user.

  • write_procs (Iterable[Union[DataProcessor[dict[str, ~DataT]], JointIndexDataProcessor[dict[str, ~DataT], ~IndexT]]]) – List of processors that will be applied to the data given by the user before writing it to the backend.

Return type

LayerSet

Returns

Layer built according to the spec.

zetta_utils.layer.volumetric.cloudvol.build_cv_layer(path, cv_kwargs=None, default_desired_resolution=None, index_resolution=None, data_resolution=None, interpolation_mode=None, readonly=False, info_reference_path=None, info_inherit_all_params=False, info_type=None, info_data_type=None, info_num_channels=None, info_chunk_size=None, info_bbox=None, info_encoding=None, info_scales=None, info_extra_scale_data=None, info_overwrite=False, info_keep_existing_scales=True, allow_slice_rounding=False, index_procs=(), read_procs=(), write_procs=())[source]

Build a CloudVolume layer.

Parameters

  • path (str) – Path to the CloudVolume.

  • cv_kwargs (UnionType[dict, None]) – Keyword arguments passed to the CloudVolume constructor.

  • default_desired_resolution (Optional[Sequence[float]]) – Default resolution used when the desired resolution is not given as a part of an index.

  • index_resolution (Optional[Sequence[float]]) – Resolution at which slices of the index will be given.

  • data_resolution (Optional[Sequence[float]]) – Resolution at which data will be read from the CloudVolume backend. When data_resolution differs from desired_resolution, data will be interpolated from data_resolution to desired_resolution using the given interpolation_mode.

  • interpolation_mode (Union[Literal[‘nearest’, ‘nearest-exact’, ‘linear’, ‘bilinear’, ‘bicubic’, ‘trilinear’, ‘area’], Literal[‘img’, ‘field’, ‘mask’, ‘segmentation’], None]) – Specification of the interpolation mode to use when data_resolution differs from desired_resolution.

  • readonly (bool) – Whether layer is read only.

  • info_reference_path (UnionType[str, None]) – Path to a reference CloudVolume for info.

  • info_scales (Optional[Sequence[Sequence[float]]]) – List of scales to be added to the info file.

  • info_type (Optional[Literal[‘image’, ‘segmentation’]]) – Type of the volume (image or segmentation).

  • info_data_type (Optional[Literal[‘uint8’, ‘int8’, ‘uint16’, ‘int16’, ‘uint32’, ‘int32’, ‘uint64’, ‘float32’]]) – Data type of the volume.

  • info_num_channels (UnionType[int, None]) – Number of channels of the volume.

  • info_chunk_size (Optional[Sequence[int]]) – Precomputed chunk size for all new scales.

  • info_bbox (UnionType[BBox3D, None]) – Bounding box corresponding to the dataset bounds for all new scales. If None, will be inherited from the reference.

  • info_encoding (UnionType[str, None]) – Precomputed encoding for all new scales.

  • info_extra_scale_data (UnionType[dict, None]) – Extra information to put into every scale. Not inherited from reference.

  • info_inherit_all_params (bool) – Whether to inherit all unspecified parameters from the reference info file. If False, only the dataset bounds will be inherited.

  • info_keep_existing_scales (bool) – Whether to keep existing scales in the info file at path.

  • info_overwrite (bool) – Whether to allow overwriting existing fields/scales in the info file at `path.

  • allow_slice_rounding (bool) – Whether layer allows IO operations where the specified index corresponds to a non-integer number of pixels at the desired resolution. When allow_slice_rounding == True, shapes will be rounded to nearest integer.

  • index_procs (Iterable[IndexProcessor[VolumetricIndex]]) – List of processors that will be applied to the index given by the user prior to IO operations.

  • read_procs (Iterable[Union[DataProcessor[ndarray[Any, dtype[+_ScalarType_co]]], JointIndexDataProcessor[ndarray[Any, dtype[+_ScalarType_co]], VolumetricIndex]]]) – List of processors that will be applied to the read data before returning it to the user.

  • write_procs (Iterable[Union[DataProcessor[UnionType[ndarray[Any, dtype[+_ScalarType_co]], Tensor]], JointIndexDataProcessor[UnionType[ndarray[Any, dtype[+_ScalarType_co]], Tensor], VolumetricIndex]]]) – List of processors that will be applied to the data given by the user before writing it to the backend.

Return type

VolumetricLayer

Returns

Layer built according to the spec.