Welcome to zetta_utils documentation!

zetta_utils is a collection of core components used in Zetta AI’s connectomics pipeline.

It includes tools for data I/O, visualization, logging, training utilities, distributed inference, and more.

Its main goal is to provide clean, powerful abstractions and accelerate connectomics research.

Getting Started

The recommended installation method is pip-installing into a virtualenv:

$ git clone  --recurse-submodules git@github.com:ZettaAI/zetta_utils.git
$ cd zetta_utils
$ ./install_zutils.py

If you want to let zetta_utils generate neuroglancer links automatically, you will need to set:

NG_STATE_SERVER_TOKEN

If you want to use Grafana integration for logging, you will need to set:

GRAFANA_CLOUD_ACCESS_KEY

Please consult the Grafana API documentation for how to set up an API key.

To use KEDA (Kubernetes Event-driven Autoscaling), you will first need to install helm.

If you are planning to use zetta_utils.viz toolkit, you will need to install nodejs:

$ curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
$ sudo apt install -y nodejs

The next steps will get you up and running in no time:

  • SubchunkableApplyFlow Quick Start Guide will introduce you to handling volumetric data in zetta_utils CUE files and running simple tasks using SubchunkableApplyFlow.

  • Examples will give you a comprehensive tour of zetta_utils’s features.

  • Module documentation will show you how each individual module is intended to be used.

  • Developer Guide will give you all of the information necessary to start contributing to zetta_utils.

  • If at any point you get confused by some terminology, please check out our Glossary.

Day-to-Day Usage

  • zetta_utils.log provides a well formated disk backed logging system.

  • zetta_utils.tensor is a unified set of operations that support both np.ndarray and torch.Tensor tensor types. Use zetta_utils.tensor_ops.unsqueeze(t) without worrying which type t is.

  • zetta_utils.layer offers flexible abstraction for data IO.

  • zetta_utils.training set of tools and integrations for neural net training.

  • zetta_utils.viz provides nifty visualization tools for Jupyter.

  • zetta_utils.builder is utility for building python objects and workflows from nested dictionaries.

  • SubchunkableApplyFlow is an inference Flow that supports arbitrary subchunking, cropping, and blending.

Project Information