Other Higher Level Tools

File Storage

XArray enables a user-friendly interface to labeling multi-dimensional arrays along with a powerful and performant backend. Therefore, we use XArray [HH17] (http://xarray.pydata.org/en/stable/) for a performant Python storage library that leverages NetCDF and promises lazy loading and incremental writes.

Simulation Management

We use MLFlow [ZCD+18] (https://mlflow.org/) for simulation management. This is typically used for managing machine-learning lifecycles but is perfectly suited for managing numerical simulations. We believe UI capability to manage simulations significantly eases the physicist’s workflow.

There are more details about how the diagnostics for a particular type of simulation are packaged and provided to the run manager object. These will be described in time. One can infer these details from the code as well.

HH17

Stephan Hoyer and Joseph J. Hamman. xarray: N-D labeled Arrays and Datasets in Python. Journal of Open Research Software, 5:1–6, apr 2017. doi:10.5334/jors.148.

ZCD+18

Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, and others. Accelerating the machine learning lifecycle with mlflow. IEEE Data Eng. Bull., 41(4):39–45, 2018.