Error in installation of `Python-3.11.5-GCCcore-13.2.0.eb` with EasyBuild 4.9.2 #21078

gkaf89 · 2024-07-28T10:23:06Z

When trying to install Python-3.11.5-GCCcore-13.2.0.eb with EasyBuild version 4.9.2, the installation fails due to an error in the installation of a file. The installation script seems to be using the log file path to store intermediate files which causes an error.

EasyBuild command:

eb --configfiles=iris_broadwell.cfg Python-3.11.5-GCCcore-13.2.0.eb

Contents of configuration file iris_broadwell.cfg:

[MAIN]
# Enable debug log mode (default: False)
#debug=
# Enable info log mode (default: False)
#info=
# Enable quiet/warning log mode (default: False)
#quiet=

[basic]
# Submit the build as a job (default: False)
job=False
# Enable dependency resolution, optionally consider additional paths to search for easyconfigs (type pathsep-separated list)
robot=/work/projects/software_stack_alpha/backup/easybuild/easyconfigs
# Additional paths to consider by robot for easyconfigs (--robot paths get priority) (type pathsep-separated list; default: /mnt/aiongpfs/projects/software_stack_alpha/backup/easybuild/easyconfigs:/work/projects/software_stack_alpha/easybuild/iris/2023b/epyc/software/EasyBuild/4.9.1/easybuild/easyconfigs)
robot-paths=%(DEFAULT_ROBOT_PATHS)s
# Additional locations to consider in --search (next to --robot and --robot-paths paths) (type pathsep-separated list)
#search-paths=

[config]
# Temporary build path (default: /home/users/gkafanas/.local/easybuild/build)
buildpath=/tmp/easybuild/iris/2023b/epyc/build
# Change prefix for buildpath, installpath, sourcepath and repositorypath (used prefix for defaults /home/users/gkafanas/.local/easybuild)
# NOTE: it seems that installpath is not determined by prefix
prefix=/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell
# Install path for software and modules (default: /home/users/gkafanas/.local/easybuild)
installpath=%(prefix)s
# Location where container recipe & image will be stored (default: /home/users/gkafanas/.local/easybuild/containers)
containerpath=%(prefix)s/containers
# The destination path for the packages built by package-tool (default: /home/users/gkafanas/.local/easybuild/packages)
packagepath=%(prefix)s/packages
# Module naming scheme to use (default: EasyBuildMNS)
module-naming-scheme=CategorizedModuleNamingScheme
# Repository path, used by repository (is passed as list of arguments to create the repository instance). For more info, use --avail-repositories. (type comma-separated list; default: /home/users/gkafanas/.local/easybuild/ebfiles_repo)
repositorypath=%(prefix)s/ebfiles_repo
# Path(s) to where sources should be downloaded (string, colon-separated) (default: /home/users/gkafanas/.local/easybuild/sources)
sourcepath=%(prefix)s/sources
# Location of Python module with hook implementations (type str)
hooks=configuration/hooks/ulhpc.py

# Directory name and format of the log file (type comma-separated tuple; default: easybuild,easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log)
logfile-format=%(prefix)s/logs/%%(name)s-%%(version)s/%%(date)s_%%(time)s,easybuild-%%(name)s-%%(version)s.log

# Backend to use for submitting jobs (type choice; default: GC3Pie) (choices: GC3Pie, PbsPython, Slurm)
job-backend=Slurm

[container]
[easyconfig]
[github]
[informative]
[job]

[override]
# Wait interval (in seconds) to use when waiting for existing lock to be removed (type <class 'int'>; default: 60)
#wait-on-lock-interval=
# Maximum amount of time (in seconds) to wait until lock is released (0 means no waiting at all, exit with error; -1 means no waiting limit, keep waiting) (type <class 'int'>; default: 0)
#wait-on-lock-limit=
[package]
[regtest]
[software]
[unittest]

Error message produced during the installation step (sub-step: == taking care of extensions...):

ERROR: Build of /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb failed (err: 'build failed (first 300 chars): Absolute path /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/%(name)s-%(version)s/20240728_023527/python passed to update_paths which only expects relative paths.')

Further details

During the failed installation sub-step the file

/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/%(name)s-%(version)s/%(date)s_%(time)s/python/sitecustomize.py

is created in the log directory (with the references in the log file path not resolved). This seems like a file that should be installed in the installation path according to the file contents:

# sitecustomize.py script installed by EasyBuild,
# to pick up Python packages installed with `--prefix` into folders listed in $EBPYTHONPREFIXES

import os
import site
import sys

# print debug messages when $EBPYTHONPREFIXES_DEBUG is defined
debug = os.getenv('EBPYTHONPREFIXES_DEBUG')

# use prefixes from $EBPYTHONPREFIXES, so they have lower priority than
# virtualenv-installed packages, unlike $PYTHONPATH

ebpythonprefixes = os.getenv('EBPYTHONPREFIXES')

if ebpythonprefixes:
    postfix = os.path.join('lib', 'python' + '.'.join(map(str, sys.version_info[:2])), 'site-packages')
    if debug:
        print("[EBPYTHONPREFIXES] postfix subdirectory to consider in installation directories: %s" % postfix)

    potential_sys_prefixes = (getattr(sys, attr, None) for attr in ("real_prefix", "base_prefix", "prefix"))
    sys_prefix = next(p for p in potential_sys_prefixes if p)
    base_paths = [p for p in sys.path if p.startswith(sys_prefix)]

    for prefix in ebpythonprefixes.split(os.pathsep):
        if debug:
            print("[EBPYTHONPREFIXES] prefix: %s" % prefix)
        sitedir = os.path.join(prefix, postfix)
        if os.path.isdir(sitedir):
            if debug:
                print("[EBPYTHONPREFIXES] adding site dir: %s" % sitedir)
            site.addsitedir(sitedir)

    # Move base python paths to the end of sys.path so modules can override packages from the core Python module
    sys.path = [p for p in sys.path if p not in base_paths] + base_paths

The installation fails when coping sitecustomize.py to its final path.

Following the installation error, the log files for Python are not created. Log files were created without issues for all packages on which Python depends.

The text was updated successfully, but these errors were encountered:

gkaf89 · 2024-07-29T08:14:05Z

Call stack produced with traceback.print_stack() at exceptions in lines 986 and 1460 of file easybuild-framework/easybuild/tools/module_generator.py:

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 789, in <module>
    main(prepared_cfg_data=(init_session_state, eb_go, cfg_settings))
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 742, in main
    do_cleanup = process_easystack(options.easystack, args, logfile, testing, init_session_state, do_build)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 289, in process_easystack
    do_cleanup &= process_eb_args([path], eb_go, cfg_settings, modtool, testing, init_session_state,
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 567, in process_eb_args
    ecs_with_res = build_and_install_software(ordered_ecs, init_session_state,
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 137, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 4264, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 4143, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 3978, in run_step
    step_method(self)()
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 2880, in extensions_step
    fake_mod_data = self.load_fake_module(purge=True, extra_modules=build_dep_mods)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 1708, in load_fake_module
    fake_mod_path = self.make_module_step(fake=True)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 3751, in make_module_step
    txt += self.make_module_extra()
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/easyblocks/p/python.py", line 599, in make_module_extra
    txt += self.module_generator.prepend_paths('PYTHONPATH', self.pythonpath)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/tools/module_generator.py", line 265, in prepend_paths
    return self.update_paths(key, paths, prepend=True, allow_abs=allow_abs, expand_relpaths=expand_relpaths)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/tools/module_generator.py", line 1461, in update_paths
    traceback.print_stack()
ERROR: Build of /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb failed (err: 'build failed (first 300 chars): Absolute path /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/   %(name)s-%(version)s/20240729_095326/python passed to update_paths which only expects relative paths.')

gkaf89 · 2024-07-29T16:01:07Z

Working back from

"/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/easyblocks/p/python.py", line 599

the problem seems to be caused by the definition of the pythonpath class member variable of the EB_Python block:

# Used for EBPYTHONPREFIXES handler script
self.pythonpath = os.path.join(log_path(), 'python')

Should pythonpath be set to the log directory? Would using build_path instead of log_path make more sense?

There are tow problems with the way the `pythonpath` instance field is defined: - it uses the log directory to store a build artifact, and - the functions that use it do not accept a full path. We cannot use the `build_path()` function instead of `log_path()` as the function provides a full path as well. Thus we opted to hard-code a relative path with respect to the build directory. Issue: easybuilders/easybuild-easyconfigs#21078

gkaf89 · 2024-07-30T08:52:30Z

Using build_path suffers from the same issue with the log_path, the function that uses the pythonpath field variable expects a relative path.

I created a pull request in the easyblocks repository: easybuilders/easybuild-easyblocks#3400

There are 2 problems with the way the `pythonpath` instance field is defined: - it uses the log directory to store a build artifact, and - the functions that use `pythonpath` do not accept a full path. We cannot use the `build_path()` function instead of `log_path()` as the function provides a full path as well. Thus we opted to hard-code a relative path with respect to the build directory. Issue: easybuilders/easybuild-easyconfigs#21078

The `pythonpath` variable in the `python.py` EasyBlock is used to create temporary directories to store build artifacts. However, there are 2 problems with the current definition of the `pythonpath` instance field: - it uses the log directory to store build artifacts, and - the functions that use `pythonpath` do not accept a full path. The `build_path()` function cannot be used instead of `log_path()` as the function provides a full path as well. Thus we opted to hard-code a relative path with respect to the build directory. Issue: easybuilders/easybuild-easyconfigs#21078

boegel · 2024-08-06T07:30:26Z

I took a quick look, and it seems like the self.pythonpath in the Python easyblock is set up assuming that the log_path() function will return a relative path, while in your case it's returning an absolute path.

With a default EasyBuild configuration, I'm seeing:

$ python3 -c "from easybuild.tools.options import set_up_configuration; from easybuild.tools.config import log_path; set_up_configuration(silent=True); print(log_path())"
easybuild

When I mimic the relevant part of your configuration though, that changes:

export EASYBUILD_LOGFILE_FORMAT='/prefix/logs/%%(name)s-%%(version)s/%%(date)s_%%(time)s,easybuild-%%(name)s-%%(version)s.log'
python3 -c "from easybuild.tools.options import set_up_configuration; from easybuild.tools.config import log_path; set_up_configuration(silent=True); print(log_path())"
/prefix/logs/%(name)s-%(version)s/%(date)s_%(time)s

The reason the Python easyblock is using log_path() is to figure out the subdirectory of the installation directory in which the EasyBuild installation log file is stored (which by default is easybuild).

So in some sense this problem is self-inflicted, it only occurs because you've customized the logfile-format configuration setting.

That said, the Python easyblock should be robust against this, of course...

boegel · 2024-08-06T07:33:55Z

After giving this some more thought, I think the directory part of logfile-format is supposed to be a path that's relative to the software installation directory. EasyBuild isn't enforcing that though (it probably should).

gkaf89 · 2024-08-06T08:44:21Z

It seems a bit restrictive the force the installation of the from log files in a path relative to the installation path.

This does not affects our case however, I can change the path to a relative path. It is already pointing inside the installation directory anyway.

gkaf89 · 2024-08-12T08:10:26Z

After giving this some more thought, I think the directory part of logfile-format is supposed to be a path that's relative to the software installation directory. EasyBuild isn't enforcing that though (it probably should).

I think we are dealing with 2 issues here.

logfile-format is expected to set a relative path, but this requirement is not documented or enforced. In my opinion it is a bit restrictive to force the log files into the directory specified by the easybuild prefix setting, but I would say this is a bug.
The pythonpath field is set to the directory containing the log files. Since the pythonpath is used to store build artifacts, I believe it is an error to store build artifacts inside a directory used to hold log files.

Am I understanding the use of pythonpath correctly? Is there a more appropriate directory to store the python artifacts?

boegel · 2024-08-14T09:45:37Z

@gkaf89 Your understanding of the pythonpath variable is correct (we should probably rename it too, because it's a bit confusing).

I think you're right that we should reconsider where the sitecustomize.py script is being stored.
That should definitely be somewhere in the installation directory.

That said, calling the easybuild subdirectory of the installation directory the "directory containing the log files" is somewhat incorrect, since other artifacts are also already stored in there, like a copy of the easyconfig file being used, patch files, the reprod directory that contains a copy of the easyblock, etc.

However, since sitecustomize.py is really part of the installation itself, it probably should be stored outside of the easybuild directory.

I guess storing it directory in the installation directory is not a good option, and then the question is in which (custom) subdirectory it should be stored (definitely not <installdir>/python/)...

Any suggestions here?

The `pythonpath` variable in the `python.py` EasyBlock is used to create temporary directories to store build artifacts. However, there are 2 problems with the current definition of the `pythonpath` instance field: - it uses the log directory to store build artifacts, and - the functions that use `pythonpath` do not accept a full path. The `build_path()` function cannot be used instead of `log_path()` as the function provides a full path as well. Thus we opted to hard-code a relative path with respect to the build directory. Issue: easybuilders/easybuild-easyconfigs#21078

The `pythonpath` variable was used to store python scripts in a temporary location and later copy them in the installation path. The scripts were stored in the path were log files are stored, `log_path()`, in both the build and installation directories. - Rename `pythonpath` to `python_script_dir_relative_path`. - Use the more appropriate location `etc/python` to store the python scripts in the build and then installation directories. Note that all functions handling the python script path accept relative paths only. Currently, only the `sitecustomize.py` python environment initialization script is stored in script directory. Issue: easybuilders/easybuild-easyconfigs#21078

gkaf89 mentioned this issue Jul 30, 2024

[bugfix] Use correct temporary directory for python installation easybuilders/easybuild-easyblocks#3398

Closed

gkaf89 mentioned this issue Jul 31, 2024

change custom easyblock for Python to set pythonpath class variable to a fixed location relative to build path easybuilders/easybuild-easyblocks#3400

Closed

boegel added the problem report label Jul 31, 2024

boegel added this to the release after 4.9.2 milestone Jul 31, 2024

boegel modified the milestones: 4.9.3, release after 4.9.3 Sep 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error in installation of `Python-3.11.5-GCCcore-13.2.0.eb` with EasyBuild 4.9.2 #21078

Error in installation of `Python-3.11.5-GCCcore-13.2.0.eb` with EasyBuild 4.9.2 #21078

gkaf89 commented Jul 28, 2024 •

edited

Loading

gkaf89 commented Jul 29, 2024

gkaf89 commented Jul 29, 2024 •

edited

Loading

gkaf89 commented Jul 30, 2024 •

edited

Loading

boegel commented Aug 6, 2024

boegel commented Aug 6, 2024

gkaf89 commented Aug 6, 2024

gkaf89 commented Aug 12, 2024

boegel commented Aug 14, 2024

Error in installation of Python-3.11.5-GCCcore-13.2.0.eb with EasyBuild 4.9.2 #21078

Error in installation of Python-3.11.5-GCCcore-13.2.0.eb with EasyBuild 4.9.2 #21078

Comments

gkaf89 commented Jul 28, 2024 • edited Loading

Further details

gkaf89 commented Jul 29, 2024

gkaf89 commented Jul 29, 2024 • edited Loading

gkaf89 commented Jul 30, 2024 • edited Loading

boegel commented Aug 6, 2024

boegel commented Aug 6, 2024

gkaf89 commented Aug 6, 2024

gkaf89 commented Aug 12, 2024

boegel commented Aug 14, 2024

Error in installation of `Python-3.11.5-GCCcore-13.2.0.eb` with EasyBuild 4.9.2 #21078

Error in installation of `Python-3.11.5-GCCcore-13.2.0.eb` with EasyBuild 4.9.2 #21078

gkaf89 commented Jul 28, 2024 •

edited

Loading

gkaf89 commented Jul 29, 2024 •

edited

Loading

gkaf89 commented Jul 30, 2024 •

edited

Loading