Skip to content
forked from wandb/sweeps

W&B Hyperparameter Sweep Engine. File sweeps related issues at the W&B client: https://github.com/wandb/client

License

Notifications You must be signed in to change notification settings

ashzblum/sweeps

 
 

Repository files navigation

W&B Hyperparameter Sweeps Engine

This repo contains the routines that generate hyperparameter sweep suggestions in the W&B backend and client local controller.

Issues are not enabled in this repository. Please open issues related to sweeps in the wandb client library github issues page.

Installation

This repository is designed to be pinned as a submodule by repos that depend on it. To install:

cd repository_that_requires_sweeps
git submodule add https://github.com/wandb/sweeps.git

To install sweeps dependencies when you install your repository's dependencies, add a line to your repository's requirements.txt that points to sweeps's requirements.txt:

-r sweeps/requirements.txt

Examples

Get next run in a sweep.

Requires two arguments, config, the config that defines the sweep, and runs, the other runs in the sweep

config:

{
    "metric": {"name": "loss", "goal": "minimize"},
    "method": "bayes",
    "parameters": {
        "v1": {"min": 1, "max": 10},
        "v2": {"min": 1.0, "max": 10.0},
    },
}

runs:

[
    SweepRun(
        name="b",
        state=RunState.finished,
        history=[
            {"loss": 5.0},
        ],
        config={"v1": {"value": 7}, "v2": {"value": 6}},
        summary_metrics={"zloss": 1.2},
    ),
    SweepRun(
        name="b2",
        state=RunState.finished,
        config={"v1": {"value": 1}, "v2": {"value": 8}},
        summary_metrics={"loss": 52.0},
        history=[],
    )
]

Codepath:

suggestion = next_run(config, runs)

next_run:

  • validates that sweep config conforms to the jsonschema in config/schema.json, if not, it raises a ValidationError
  • parses the config file and determines the method that it should use to find the next run (in this case bayes_search_next_run)
  • calls bayes_search_next_run(config, runs) and returns the suggested SweepRun

Return list of runs to stop in a sweep.

Requires two arguments, config, the config that defines the sweep, and runs, the other runs in the sweep

config:

{
    "method": "grid",
    "metric": {"name": "loss", "goal": "minimize"},
    "early_terminate": {
        "type": "hyperband",
        "max_iter": 5,
        "eta": 2,
        "s": 2,
    },
    "parameters": {"a": {"values": [1, 2, 3]}},
}

runs:

[
    SweepRun(
        name="a",
        state=RunState.finished,  # This is already stopped
        history=[
            {"loss": 10},
            {"loss": 9},
        ],
    ),
    SweepRun(
        name="b",
        state=RunState.running,  # This should be stopped
        history=[
            {"loss": 10},
            {"loss": 10},
        ],
    ),
    SweepRun(
        name="c",
        state=RunState.running,  # This passes band 1 but not band 2
        history=[
            {"loss": 10},
            {"loss": 8},
            {"loss": 8},
        ],
    ),
    SweepRun(
        name="d",
        state=RunState.running,
        history=[
            {"loss": 10},
            {"loss": 7},
            {"loss": 7},
        ],
    ),
    SweepRun(
        name="e",
        state=RunState.finished,
        history=[
            {"loss": 10},
            {"loss": 6},
            {"loss": 6},
        ],
    ),
]

Codepath:

to_stop = stop_runs(config, runs)

stop_runs:

  • validates that sweep config conforms to the jsonschema in config/schema.json, if not, it raises a ValidationError
  • parses the config file and determines the method that it should use to early terminate runs (in this case hyperband_stop_runs)
  • calls hyperband_stop_runs(config, runs) and returns the SweepRuns to stop

Testing

To run tests:

pip install -r requirements.dev.txt
pytest

Contributing

Install the development requirements:

pip install -r requirements.dev.txt

Install the pre-commit hooks:

pre-commit install .

PRs must:

  • Not degrade test coverage (automatically calculated via codecov)
  • Use type-hints on all public functions
  • Pass linting checks
  • Be reviewed and approved by at least 1 maintainer

About

W&B Hyperparameter Sweep Engine. File sweeps related issues at the W&B client: https://github.com/wandb/client

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%