Features

• Run Table Model: Framework support to easily define an experiment's measurements with Factors, their Treatment levels, exclude certain combinations of Treatments, and add data columns for storing aggregated data.
• Restarting: If an experiment was not entirely completed on the last invocation (e.g. some variations crashes), experiment runner can be re-invoked to finish any remaining experiment variations.
• Persistency: Raw and aggregated experiment data per variation can be persistently stored.
• Operational Types: Two operational types: AUTO and SEMI, for more fine-grained experiment control.
• Progress Indicator: Keeps track of the execution of each run of the experiment
• Target and profiler agnostic: Can be used with any target to measure (e.g. ELF binary, .apk over adb, etc.) and with any profiler (e.g. WattsUpPro, etc.)

Experiment events

When a user experiment is run, the following list of events are raised in order automatically by Experiment Runner:

• BEFORE_EXPERIMENT - Invoked only once.
• For each variation, the following events are raised in order:
  1. BEFORE_RUN Invoked before each variation
  2. START_RUN
  3. START_MEASUREMENT
  4. INTERACT
  5. CONTINUE - Only to be used by OperationType.SEMI configs. (Not automatically subscribed to by the generated config.)
  6. STOP_MEASUREMENT
  7. STOP_RUN
  8. POPULATE_RUN_DATA
  9. Wait for RunnerConfig.time_between_runs_in_ms milliseconds
• AFTER_EXPERIMENT - Invoked only once.

TODO: Add visualization similar to robot-runner timeline of events

Experiment variations

Variations are automatically created by the Experiment Runner in accordance to the user-defined run table (Factors, Treatment levels, and variation exclusions).

Further detailed description of the events and their expected callback behavior can be found in the generated config. One thing to notice in the config is that, each callback function that is associated with a variation, accepts a context: RunnerContext parameter that describes the current variation.