Internal Details

The framework offers an automation of the infrastructure overhead for measurement-based empirical experiments, as a consequence of its design, produced by the following design drivers:

• User Authority: Give the user full authority over the experiment execution in the Python-based configuration file.
• Focus on Orchestration: Orchestrate the experiment on the basis of events. These events are characterized by their moment of execution in any experiment.
• Focus on Supporting Infrastructure: Offer the user all, potentially necessary, supporting features (e.g. factors and treatment levels).

Experiment Runner consists of the following components:

• Experiment orchestrator: Is in charge of executing the whole experiment according to the experiment configuration provided by the user.
• Event manager: Provides the user with subscribable events, to which callback methods can be set, which are called at the appropriate time by the Experiment Orchestrator.
• Progress manager: Keeps track of the execution of each run of the experiment.
• Config Validator: Provides a validation of a user's configuration file and checks system readiness.

TODO: Add visualization similar to robot-runner overview

When Experiment Runner is passed a user-defined experiment config via command line arguments, it will:

• Validate the config
• Output the config's values as read by Experiment Runner in the console for user validation
• Create the experiment folder
• Create the run table (.csv), and persist it in the experiment folder
• Execute the experiment on a per-variation basis, going over each variation with its specified treatments in the run table.