scida: scalable analysis for scientific big data
Chris Byrohl, Dylan Nelson
JOSS · October 2023
Abstract
scida is a Python package for reading and analyzing large scientific datasets, with out-of-the-box support for various cosmological and galaxy formation simulations. Data access is provided through a hierarchical, dictionary-like data structure after a simple load() function. Using the dask library for scalable, parallel, and out-of-core computation, all computation requests from a user session are first collected in a task graph. Arbitrary custom analysis, as well as all available dask array operations, can be performed. Computation is executed only upon request on a target resource, such as an HPC cluster.
scida is a Python package for scalable analysis of large scientific datasets. It provides a unified interface for reading and analyzing data from various astrophysical simulations and observations, with built-in support for distributed computing via dask.
Key Features
- Unified Interface: Read data from AREPO, GADGET, SWIFT, GIZMO simulations and observational datasets with a consistent API
- Lazy Evaluation: Data is only loaded when needed, enabling work with datasets larger than memory
- Scalable: Built on dask for parallel and distributed computing
- Unit-Aware: Automatic unit handling and conversion via pint
- Extensible: Easy to add support for new data formats
Visual Impressions
scida can be used to analyze and visualize a wide variety of astrophysical datasets.
Cosmological Simulations
Observational Data
Links
- Documentation: scida.io
- Source Code: github.com/cbyrohl/scida
- PyPI: pypi.org/project/scida