Welcome to `zetta_utils` documentation!¶

zetta_utils is a collection of core components used in Zetta AI’s connectomics pipeline.

It includes tools for data I/O, visualization, logging, training utilities, distributed inference, and more.

Its main goal is to provide clean, powerful abstractions and accelerate connectomics research.

Getting Started¶

The recommended installation method is pip-installing into a virtualenv:

$ git clone  --recurse-submodules git@github.com:ZettaAI/zetta_utils.git
$ cd zetta_utils
$ ./install_zutils.py

If you want to let zetta_utils generate neuroglancer links automatically, you will need to set:

NG_STATE_SERVER_TOKEN

If you want to use Grafana integration for logging, you will need to set:

GRAFANA_CLOUD_ACCESS_KEY

Please consult the Grafana API documentation for how to set up an API key.

To use KEDA (Kubernetes Event-driven Autoscaling), you will first need to install helm.

If you are planning to use zetta_utils.viz toolkit, you will need to install nodejs:

$ curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
$ sudo apt install -y nodejs

The next steps will get you up and running in no time:

SubchunkableApplyFlow Quick Start Guide will introduce you to handling volumetric data in zetta_utils CUE files and running simple tasks using SubchunkableApplyFlow.
Examples will give you a comprehensive tour of zetta_utils’s features.
Module documentation will show you how each individual module is intended to be used.
Developer Guide will give you all of the information necessary to start contributing to zetta_utils.
If at any point you get confused by some terminology, please check out our Glossary.

zetta_utils.log provides a well formated disk backed logging system.
zetta_utils.tensor is a unified set of operations that support both np.ndarray and torch.Tensor tensor types. Use zetta_utils.tensor_ops.unsqueeze(t) without worrying which type t is.
zetta_utils.layer offers flexible abstraction for data IO.
zetta_utils.training set of tools and integrations for neural net training.
zetta_utils.viz provides nifty visualization tools for Jupyter.
zetta_utils.builder is utility for building python objects and workflows from nested dictionaries.
SubchunkableApplyFlow is an inference Flow that supports arbitrary subchunking, cropping, and blending.