The Sacred Infrastructure for Computational Research

We present a toolchain for computational research consisting of Sacred and two supporting tools. Sacred is an open source Python framework which aims to provide basic infrastructure for running computational experiments independent of the methods and libraries used. Instead, it focuses on solving universal everyday problems, such as managing configurations, reproducing results, and bookkeeping. Moreover, it provides an extensible basis for other tools, two of which we present here: Labwatch helps with tuning hyperparameters, and Sacredboard provides a web-dashboard for organizing and analyzing runs

[1]  Ralph Johnson,et al.  Design patterns: elements of reuseable object-oriented software , 1994 .

[2]  Computing In Science & Engineering: Web Computing - Java and Grande Applications , 2003, IEEE Distributed Syst. Online.

[3]  Bertram Ludäscher,et al.  Kepler: an extensible system for design and execution of scientific workflows , 2004, Proceedings. 16th International Conference on Scientific and Statistical Database Management, 2004..

[4]  Cláudio T. Silva,et al.  VisTrails: enabling interactive multiple-view visualizations , 2005, VIS 05. IEEE Visualization, 2005..

[5]  Daniel J. Blankenberg,et al.  Galaxy: a platform for interactive large-scale genome analysis. , 2005, Genome research.

[6]  Geoff Holmes,et al.  Experiment databases , 2012, Machine Learning.

[7]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[8]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[9]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[10]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[11]  Philip J. Guo.  CDE: A Tool for Creating Portable Experimental Software Packages , 2012, Computing in Science & Engineering.

[12]  Bill Howe.  CDE: A Tool for Creating Portable Experimental Software Packages , 2012 .

[13]  Andrew P. Davison.  Automated Capture of Experiment Context for Easier Reproducibility in Computational Research , 2012, Computing in Science & Engineering.

[14]  Andreas Wombacher,et al.  ProvenanceCurious: a tool to infer data provenance from scripts , 2013, EDBT '13.

[15]  Katharina Eggensperger,et al.  Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters , 2013 .

[16]  Ian T. Foster,et al.  Using Provenance for Repeatability , 2013, TaPP.

[17]  Carole A. Goble,et al.  The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud , 2013, Nucleic Acids Res..

[18]  Leslie Greengard,et al.  Fast Direct Methods for Gaussian Processes and the Analysis of NASA Kepler Mission Data , 2014 .

[19]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[20]  Tony R. Martinez,et al.  An Easy to Use Repository for Comparing and Improving Machine Learning Algorithm Usage , 2014, MetaSel@ECAI.

[21]  Kevin Leyton-Brown,et al.  An Efficient Approach for Assessing Hyperparameter Importance , 2014, ICML.

[22]  Yves Janin,et al.  CARE, the comprehensive archiver for reproducible execution , 2014, TRUST '14.

[23]  Juliana Freire,et al.  Collecting and Analyzing Provenance on Interactive Notebooks: When IPython Meets noWorkflow , 2015, TaPP.

[24]  Prabhat,et al.  Scalable Bayesian Optimization Using Deep Neural Networks , 2015, ICML.

[25]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[26]  Aaron Klein,et al.  Bayesian Optimization with Robust Bayesian Neural Networks , 2016, NIPS.

[27]  Dennis Shasha,et al.  ReproZip: Computational Reproducibility With Ease , 2016, SIGMOD Conference.

[28]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Vasan Subramanian,et al.  MongoDB , 2019, Pro MERN Stack.