No module named 'dask.compatibility' for autogluon docker image

[SubhadityaMukherjee]

Hey, I have been trying to build the docker image for autogluon for a while now but I keep running into missing requirements.
The current one seems to be missing dask.compatibility. I tried installing dask[complete] but maybe I did something wrong. (I did update my repo to the new one)
Any ideas?

Im running it on a Mac with the M series chip.

#21 57.86  + zict==3.0.0
#21 57.86 warning: The package `autogluon-tabular==0.1.0` does not have an extra named `skex`
#21 57.86 Traceback (most recent call last):
#21 57.86   File "<string>", line 1, in <module>
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/autogluon/tabular/__init__.py", line 3, in <module>
#21 57.86     from autogluon.core.dataset import TabularDataset
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/autogluon/core/__init__.py", line 6, in <module>
#21 57.86     from .scheduler.resource.resource import *
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/autogluon/core/scheduler/__init__.py", line 1, in <module>
#21 57.86     from .import remote, resource
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/autogluon/core/scheduler/remote/__init__.py", line 2, in <module>
#21 57.86     from .remote import *
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/autogluon/core/scheduler/remote/remote.py", line 11, in <module>
#21 57.86     from distributed import Client
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/distributed/__init__.py", line 7, in <module>
#21 57.86     from .actor import Actor, ActorFuture
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/distributed/actor.py", line 6, in <module>
#21 57.86     from .client import Future, default_client
#21 57.86   File "/bench/frameworks/AutoGluon/venv/lib/python3.9/site-packages/distributed/client.py", line 30, in <module>
#21 57.86     from dask.compatibility import apply
#21 57.86 ModuleNotFoundError: No module named 'dask.compatibility'
#21 57.86 
#21 57.86 Command '['/bench/frameworks/AutoGluon/setup.sh', 'stable']' returned non-zero exit status 1.
#21 57.86 Traceback (most recent call last):
#21 57.86   File "/bench/runbenchmark.py", line 204, in <module>
#21 57.86     bench.setup(amlb.SetupMode[args.setup])
#21 57.86   File "/bench/amlb/benchmark.py", line 151, in setup
#21 57.86     self.framework_module.setup(*self.framework_def.setup_args,
#21 57.86   File "/bench/frameworks/AutoGluon/__init__.py", line 9, in setup
#21 57.86     call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
#21 57.86   File "/bench/amlb/utils/process.py", line 293, in call_script_in_same_dir
#21 57.86     return run_script(script_path, *args, **kwargs)
#21 57.86   File "/bench/amlb/utils/process.py", line 287, in run_script
#21 57.86     return run_cmd(script_path, *args, **kwargs)
#21 57.86   File "/bench/amlb/utils/process.py", line 281, in run_cmd
#21 57.86     raise e
#21 57.86   File "/bench/amlb/utils/process.py", line 255, in run_cmd
#21 57.86     completed = run_subprocess(str_cmd if params.shell else full_cmd,
#21 57.86   File "/bench/amlb/utils/process.py", line 98, in run_subprocess
#21 57.86     raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
#21 57.86 subprocess.CalledProcessError: Command '['/bench/frameworks/AutoGluon/setup.sh', 'stable']' returned non-zero exit status 1.
#21 ERROR: process "/bin/sh -c $PY runbenchmark.py autogluon -s only" did not complete successfully: exit code: 2
------
 > [17/17] RUN /bench/venv/bin/python3.9 -W ignore runbenchmark.py autogluon -s only:
57.86     return run_script(script_path, *args, **kwargs)
57.86   File "/bench/amlb/utils/process.py", line 287, in run_script
57.86     return run_cmd(script_path, *args, **kwargs)
57.86   File "/bench/amlb/utils/process.py", line 281, in run_cmd
57.86     raise e
57.86   File "/bench/amlb/utils/process.py", line 255, in run_cmd
57.86     completed = run_subprocess(str_cmd if params.shell else full_cmd,
57.86   File "/bench/amlb/utils/process.py", line 98, in run_subprocess
57.86     raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
57.86 subprocess.CalledProcessError: Command '['/bench/frameworks/AutoGluon/setup.sh', 'stable']' returned non-zero exit status 1.
------
Dockerfile:50
--------------------
  48 |     RUN (grep -v '^\s*#' | xargs -L 1 $PIP install --no-cache-dir) < requirements.txt
  49 |     
  50 | >>> RUN $PY runbenchmark.py autogluon -s only
  51 |     
  52 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c $PY runbenchmark.py autogluon -s only" did not complete successfully: exit code: 2

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/j2x2yeijf40ywm1myo12ajfzz

Command 'docker build   -t automlbenchmark/autogluon:stable-dev -f /Users/smukherjee/Documents/CODE/Github/automlbenchmark/frameworks/AutoGluon/.setup/Dockerfile .' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/Users/smukherjee/Documents/CODE/Github/automlbenchmark/runbenchmark.py", line 204, in <module>
    bench.setup(amlb.SetupMode[args.setup])
  File "/Users/smukherjee/Documents/CODE/Github/automlbenchmark/amlb/runners/container.py", line 82, in setup
    self.image = self._build_image(cache=(mode != SetupMode.force))
  File "/Users/smukherjee/Documents/CODE/Github/automlbenchmark/amlb/runners/container.py", line 197, in _build_image
    self._run_container_build_command(image, cache)
  File "/Users/smukherjee/Documents/CODE/Github/automlbenchmark/amlb/runners/docker.py", line 100, in _run_container_build_command
    run_cmd("docker build {options} {build_extra_options} -t {container} -f {script} .".format(
  File "/Users/smukherjee/Documents/CODE/Github/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd
    raise e
  File "/Users/smukherjee/Documents/CODE/Github/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/Users/smukherjee/Documents/CODE/Github/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'docker build   -t automlbenchmark/autogluon:stable-dev -f /Users/smukherjee/Documents/CODE/Github/automlbenchmark/frameworks/AutoGluon/.setup/Dockerfile .' returned non-zero exit status 1.

[PGijsbers]

Please include the version of automl benchmark you are running (e.g., commit hash, branch), your Python version, and the exact command you used to invoke the benchmark. Thanks!

[SubhadityaMukherjee]

Ah good point.
I am using commit ac09497 (HEAD -> master, origin/master, origin/HEAD)
(Add SapientML to automl benchmark (#630))

I am using Python 3.10.14
Docker version 26.0.0, build 2ae903e

The command used is python3 runbenchmark.py autogluon openml/t/10 --mode docker

Is this enough information? If not, I can add more details. :)

[PGijsbers]

First, try using Python 3.9. It's the only officially supported version of the benchmark. I've often encountered issues with dependencies using other Python versions. It's something we hope to address in the future.

[SubhadityaMukherjee]

No, it still does not work ):