The Chronos System is a comprehensive evaluation toolkit initially conceived for database systems evaluations but designed generically enough to be applicable in other domains. Its primary goal is to streamline and automate the system evaluation process, including setup, execution, and subsequent analysis, thereby saving significant amounts of time and eliminating manual tasks.

Systems evaluations are an integral part of empirical research in computer science, involving the systematic assessment of systems based on various parameters. These evaluations often require systems to be run repeatedly with different parameter sets, which can be a time-consuming and repetitive task. The Chronos system revolutionizes this process by automating the entire evaluation workflow, requiring no human intervention once set up.

The Chronos system offers several key features that make it an efficient and effective tool for system evaluations. It provides an easy-to-use interface for defining new experiments, scheduling their execution, monitoring progress, and analyzing results. The system can handle different systems simultaneously and is designed to support long-term evaluations by providing automated failure handling and recovery mechanisms. Results from evaluations, including parameter settings that led to those results, are stored and archived within the system for future reference. Moreover, Chronos facilitates the integration of existing evaluation clients and supports developers in creating new ones.

The architecture of the Chronos system consists of two main components: Chronos Control and Chronos Agents. Chronos Control is the heart of the toolkit, providing a web UI and a RESTful API for managing evaluations. On the other hand, Chronos Agents interact with existing evaluation clients and leverage the REST API to obtain necessary information for evaluations. Chronos also has a built-in data model containing elements such as projects, experiments, evaluations, jobs, systems, and deployments, making it easier to manage and navigate the evaluation process.

Since its inception, Chronos has been used for the evaluation of various research prototypes and has proven itself as an invaluable tool in academic and research settings. Its capacity for automation, coupled with its robust and adaptable features, positions Chronos as a vital tool for systematic systems evaluations across different domains. With the source code openly available on GitHub under the MIT open-source license, the Chronos system represents a significant contribution to empirical research in computer science