Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance generic Bundle easyblock to transfer module requirements of components #3472

Merged

Conversation

Thyre
Copy link
Contributor

@Thyre Thyre commented Oct 7, 2024

This PR implements the make_module_req_guess method for the generic Bundle EasyBlock. With this, all the requirements of the components in a bundle are transferred correctly to the final module. Previously, this could lead to missing environment variables, letting the build succeed but still resulting in a broken module, for example because PATH is not set.

Fixes #2733

@Thyre
Copy link
Contributor Author

Thyre commented Oct 7, 2024

I've only tested the changes with easybuilders/easybuild-easyconfigs#21582 so far. I'll try to gather the EasyConfigs using Bundle in some way to make sure that I did not break them, since this is a change on a widely used EasyBlock.

Unfortunately, I can only test GCC 12.3.0 and newer easily and are somewhat limited with my compute resources. So this might take a while..

@Thyre
Copy link
Contributor Author

Thyre commented Oct 7, 2024

Example with the changed EasyBlock:

easybuilders/easybuild-easyconfigs#21582 (comment)

@Thyre
Copy link
Contributor Author

Thyre commented Oct 7, 2024

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS FlexiBLAS-3.4.4-GCC-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/a33ae670ab1932bdcfff62bae82c3f53 for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Oct 7, 2024

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS GCC-13.3.0.eb
  • SUCCESS Autotools-20231222-GCCcore-13.3.0.eb
  • SUCCESS cffi-1.16.0-GCCcore-13.3.0.eb
  • SUCCESS Julia-1.9.2-linux-x86_64.eb
  • SUCCESS groff-1.23.0-GCCcore-13.3.0.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS expat-2.6.2-GCCcore-13.3.0.eb
  • SUCCESS libiconv-1.17-GCCcore-13.3.0.eb
  • SUCCESS patchelf-0.18.0-GCCcore-13.3.0.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gettext-0.22.5-GCCcore-13.3.0.eb
  • SUCCESS Rust-1.78.0-GCCcore-13.3.0.eb
  • SUCCESS intltool-0.51.0-GCCcore-13.3.0.eb
  • SUCCESS Doxygen-1.11.0-GCCcore-13.3.0.eb
  • SUCCESS hatchling-1.24.2-GCCcore-13.3.0.eb
  • SUCCESS setuptools-rust-1.9.0-GCCcore-13.3.0.eb
  • SUCCESS maturin-1.6.0-GCCcore-13.3.0.eb
  • SUCCESS typing-extensions-4.11.0-GCCcore-13.3.0.eb
  • FAIL (build issue) pydantic-2.9.1-GCCcore-13.3.0.eb (partial log available at https://gist.github.com/Thyre/abd4413683d3801209f2daa0a6d77ed8)
  • SUCCESS gperf-3.1-GCCcore-13.3.0.eb
  • SUCCESS util-linux-2.40-GCCcore-13.3.0.eb
  • SUCCESS libpng-1.6.43-GCCcore-13.3.0.eb
  • SUCCESS Brotli-1.1.0-GCCcore-13.3.0.eb
  • SUCCESS freetype-2.13.2-GCCcore-13.3.0.eb
  • SUCCESS fontconfig-2.15.0-GCCcore-13.3.0.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 25 out of 26 (7 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/3b8848b109d905611283a882db8bb378 for a full test report.


The failed one doesn't look related:

Building a mixed python/rust project
       Found pyo3 bindings
       maturin failed
        Caused by: Python interpreter should be a kind of interpreter (e.g. 'python3.8' or 'pypy3.9') when cross-compiling, got path to interpreter: /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/eb-1tc417di/pip-modern-metadata-s3zbqxe6', '--interpreter', '/tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/eb-1tc417di/pip-modern-metadata-s3zbqxe6 --interpreter /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python`
      [end of output]

The tested ones include normal Bundle (e.g. GCC & Autotools), PerlBundle (Perl-bundle-CPAN), PythonBundle (cffi) and JuliaBundle (Circuitscape). CargoPythonBundle failed due to the failure shown above.

@Thyre
Copy link
Contributor Author

Thyre commented Oct 11, 2024

Changed PR so that the EasyBlock for each component is stored and not initialized in multiple steps.

@Thyre
Copy link
Contributor Author

Thyre commented Oct 11, 2024

Based on https://docs.easybuild.io/version-specific/easyblocks/?h=bundle, this change should affect the following EasyBlocks:

  • Bundle
  • BuildEnv
  • CrayToolchain
  • EB_Clang_minus_AOMP
  • EB_OpenSSL_wrapper
  • JuliaBundle
  • PerlBundle
  • PythonBundle
  • CargoPythonBundle
  • SystemCompiler
  • SystemMPI
  • Toolchain

I will not be able to test all of them (e.g. CrayToolchain), but I'll try to get one EasyConfig for all of these EasyBlocks to make sure that nothing broke and compare their modules.

@Thyre
Copy link
Contributor Author

Thyre commented Oct 25, 2024

I've chosen a few EasyConfigs to compare with this MR and the current develop.

Building will take quite some time. I'll post the resulting module diffs later.

@Thyre
Copy link
Contributor Author

Thyre commented Oct 26, 2024

Now that all modules are finally installed (but I had to partially select other modules due to general installation issues), I found one bug with the proposed changes:

When checking crypt4gh-1.7-GCC-12.3.0.eb, the module does a sanity check for an executable in /bin. This path is correctly added in develop:

help([==[

Description
===========
crypt4gh is a Python tool to encrypt, decrypt or re-encrypt files, 
according to the GA4GH encryption file format.


More information
================
 - Homepage: https://github.com/EGA-archive/crypt4gh


Included extensions
===================
crypt4gh-1.7
]==])

whatis([==[Description: crypt4gh is a Python tool to encrypt, decrypt or re-encrypt files, 
according to the GA4GH encryption file format.]==])
whatis([==[Homepage: https://github.com/EGA-archive/crypt4gh]==])
whatis([==[URL: https://github.com/EGA-archive/crypt4gh]==])
whatis([==[Extensions: crypt4gh-1.7]==])

local root = "/tank/Programs/Linux/EasyBuild/datenlager/Software/software/crypt4gh/1.7-GCC-12.3.0"

conflict("crypt4gh")

if not ( isloaded("GCC/12.3.0") ) then
    load("GCC/12.3.0")
end

if not ( isloaded("Python/3.11.3-GCCcore-12.3.0") ) then
    load("Python/3.11.3-GCCcore-12.3.0")
end

if not ( isloaded("Python-bundle-PyPI/2023.06-GCCcore-12.3.0") ) then
    load("Python-bundle-PyPI/2023.06-GCCcore-12.3.0")
end

if not ( isloaded("PyYAML/6.0-GCCcore-12.3.0") ) then
    load("PyYAML/6.0-GCCcore-12.3.0")
end

if not ( isloaded("cryptography/41.0.1-GCCcore-12.3.0") ) then
    load("cryptography/41.0.1-GCCcore-12.3.0")
end

if not ( isloaded("bcrypt/4.0.1-GCCcore-12.3.0") ) then
    load("bcrypt/4.0.1-GCCcore-12.3.0")
end

prepend_path("CMAKE_PREFIX_PATH", root)
prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("PATH", pathJoin(root, "bin"))
setenv("EBROOTCRYPT4GH", root)
setenv("EBVERSIONCRYPT4GH", "1.7")
setenv("EBDEVELCRYPT4GH", pathJoin(root, "easybuild/crypt4gh-1.7-GCC-12.3.0-easybuild-devel"))

prepend_path("PYTHONPATH", pathJoin(root, "lib/python3.11/site-packages"))
-- Built with EasyBuild version 4.9.5.dev0
setenv("EBEXTSLISTCRYPT4GH", "crypt4gh-1.7")

but this fails with the proposed changes. CMAKE_PREFIX_PATH, LIBRARY_PATH and PATH are missing. I suspect that adding make_module_req_guess to Bundle causes this to fail.

@Thyre Thyre force-pushed the generic-bundle-include-module-requirements branch from 07b4490 to 24cec43 Compare October 26, 2024 17:14
@Thyre
Copy link
Contributor Author

Thyre commented Oct 26, 2024

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS GCC-system.eb
  • SUCCESS OpenMPI-system-GCC-system-2.29.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/75aa472a40750fd854cf2d3ec2ede6e3 for a full test report.


I'm not able to test jiter-0.4.1-GCCcore-12.3.0.eb due to the following error caused by maturin:

  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

   Preparing metadata (pyproject.toml) did not run successfully.
   exit code: 1
  > [6 lines of output]
       Found pyo3 bindings
       maturin failed
        Caused by: Python interpreter should be a kind of interpreter (e.g. 'python3.8' or 'pypy3.9') when cross-compiling, got path to interpreter: /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/eb-ibucduig/pip-modern-metadata-debc3bzs', '--interpreter', '/tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/eb-ibucduig/pip-modern-metadata-debc3bzs --interpreter /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python`
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

 Encountered error while generating package metadata.
> See above for output.

I've also checked if the module files of the EasyConfigs above changed. They did not.

@Thyre Thyre force-pushed the generic-bundle-include-module-requirements branch from 24cec43 to 5e3d7e4 Compare October 26, 2024 18:47
@Thyre
Copy link
Contributor Author

Thyre commented Oct 26, 2024

Full (re-)installation

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS GCC-system.eb
  • SUCCESS OpenMPI-system-GCC-system-2.29.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/bbc3468bc740b0c511c9f1e665d32910 for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Oct 26, 2024

Module only

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS GCC-system.eb
  • SUCCESS OpenMPI-system-GCC-system-2.29.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/c78da935cdd54f14102e4d932295777e for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Oct 27, 2024

Testing a few of the Stages 2025 EasyConfigs using Bundle/PythonBundle/... of the JSC repo

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS Autotools-20231222-GCCcore-13.3.0.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS boto3-1.35.36-GCCcore-13.3.0.eb
  • SUCCESS npsmpic-2024a.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
jrlogin05.jureca - Linux Rocky Linux 9.4, x86_64, AMD EPYC 7742 64-Core Processor, 2 x NVIDIA Quadro RTX 8000, 560.35.03, Python 3.9.18
See https://gist.github.com/Thyre/81e26fbe93c25adabffd33e5664ddc0b for a full test report.

@boegel boegel changed the title Enhance generic Bundle EasyBlock to transfer module requirements of components Enhance generic Bundle easyblock to transfer module requirements of components Nov 6, 2024
@Thyre
Copy link
Contributor Author

Thyre commented Nov 6, 2024

Bi-weekly EasyBuild conf call 2024-11-06: While these changes continue to work with EasyBuild 5.0.x, the make_module_req_guess method will be deprecated.

My current idea would be to continue with this PR as it is. Once easybuilders/easybuild-framework#4653 is merged, I'll work on porting these changes to the new method for EasyBuild 5.0.x.

@Thyre
Copy link
Contributor Author

Thyre commented Nov 8, 2024

@boegelbot please test @ jsc-zen3
EB_ARGS="X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"
CORE_CNT=16

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5224

Test results coming soon (I hope)...

- notification for comment with ID 2464270943 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS X11-20240607-GCCcore-13.3.0.eb
  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
  • SUCCESS GCC-system.eb
  • SUCCESS gfbf-2024a.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/f0ee3fa8737148ccb5fae372b743d256 for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Nov 8, 2024

@boegelbot please test @ jsc-zen3
EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5226

Test results coming soon (I hope)...

- notification for comment with ID 2464420612 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS X11-20240607-GCCcore-13.3.0.eb
  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
  • SUCCESS GCC-system.eb
  • SUCCESS gfbf-2024a.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/b951173c327e4bb0f8b38422bdf91456 for a full test report.

Copy link
Contributor

@lexming lexming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I only have a few comments to improve the code a bit. See below

easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved
easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved
@Thyre
Copy link
Contributor Author

Thyre commented Nov 8, 2024

Thanks a lot for the suggestions. I'll add them to the commit in a moment 😄

…omponents

This commit implements the `make_module_req_guess` method for the
generic Bundle EasyBlock. With this, all the requirements of the
components in a bundle are transferred correctly to the final module.
Previously, this could lead to missing environment variables, letting
the build succeed but still resulting in a broken module, for example
because `PATH` is not set.

Signed-off-by: Jan André Reuter <[email protected]>
@Thyre Thyre force-pushed the generic-bundle-include-module-requirements branch 2 times, most recently from 8247d63 to 957ee53 Compare November 8, 2024 13:31
Instead of zipping bundles and configs every time, use a single instance list.
Also, do not check for the type when building the module requirements and
catch the exception instead.

Signed-off-by: Jan André Reuter <[email protected]>
@Thyre Thyre force-pushed the generic-bundle-include-module-requirements branch from 957ee53 to eda3ca7 Compare November 8, 2024 13:31
@Thyre
Copy link
Contributor Author

Thyre commented Nov 8, 2024

@boegelbot please test @ jsc-zen3
EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5235

Test results coming soon (I hope)...

- notification for comment with ID 2464855921 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS X11-20240607-GCCcore-13.3.0.eb
  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
  • SUCCESS GCC-system.eb
  • SUCCESS gfbf-2024a.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/2df274b1d2731957ac6b0e5be8d36daf for a full test report.

@lexming
Copy link
Contributor

lexming commented Nov 8, 2024

@Thyre thanks for the quick update! do you want to add your name to the list of contributors to this easyblock? If you do, there is no need to send new tests again 🙂

Signed-off-by: Jan André Reuter <[email protected]>
@Thyre
Copy link
Contributor Author

Thyre commented Nov 8, 2024

@Thyre thanks for the quick update! do you want to add your name to the list of contributors to this easyblock?

Done. Took longer than I want to admit to find the authors comment 😅

If you do, there is no need to send new tests again 🙂

Agreed, changing the comment shouldn't break anything. Just wanted to make sure nothing broke during the last changes.

Copy link
Contributor

@lexming lexming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lexming
Copy link
Contributor

lexming commented Nov 9, 2024

Merging, thanks @Thyre !

@lexming lexming merged commit 27c471c into easybuilders:develop Nov 9, 2024
41 checks passed
# list of EasyConfig instances for components
self.comp_cfgs = []
# list of EasyConfig instances and their EasyBlocks for components
self.comp_instances = []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Thyre renaming this will have some fallout, see the clang_aomp.py where self.comp_cfgs is used...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, we should keep the old one, yeah. With the PR reverted, I revise the changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah... I missed that EasyBlock since it is unused. However, there is only a single usage of self.comp_cfgs.
When I open a new PR for this, I'll update the clang_aomp.py EasyBlock as well. Haven't found other usages.

@boegel
Copy link
Member

boegel commented Nov 12, 2024

@Thyre @lexming I think the changes here may lead to an infinite loop, the easyconfigs test suite is failing due to generating too many open files, and it seems to coincide with this PR getting merged...


for cfg, comp in self.comp_instances:
self.log.info("Gathering module paths for component %s v%s", cfg['name'], cfg['version'])
reqs = comp.make_module_req_guess()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if one of the component itself is a Bundle instance, that may result in an infinite loop I think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... a Bundle instance should gather the requirements of the components. If one of the components is also a Bundle, we should collect all of them recursively.

An infinite loop can be achieved if one EasyBlock tries to call this function via super. Can we prevent that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@boegel I don't see why there would be an infinite loop here, the list of components is finite and equal to whatever the easyconfig defines. Even if there is a bundle in the component list, it is also defined in the easyconfig.
This is not like looping over dependencies, with multiple layers of depth.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a Bundle as component of another Bundle is explicitly forbidden. See

raise EasyBuildError("The Bundle easyblock can not be used to install components in a bundle")

So this is not the cause of the issue (unless some unit test does some weird stuff and hijacks that check in Bundle)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently there is a general bug in this check, as it does not cover easyblocks based on Bundle, like PythonBundle or OpenSSL_wrapper. So that is one thing that can improve.
However, it is already impossible to make a nested bundle in practice because the resulting construct always errors out. If the bundle component has sources, it results in error about "bundles cannot have sources"; and if it does not have sources it results in error about "missing sources".

boegel added a commit to boegel/easybuild-easyblocks that referenced this pull request Nov 12, 2024
@boegel
Copy link
Member

boegel commented Nov 12, 2024

@Thyre @lexming I think the changes here may lead to an infinite loop, the easyconfigs test suite is failing due to generating too many open files, and it seems to coincide with this PR getting merged...

I've opened #3504 to revert the changes that were done here.
Purely based on timing, it seems like the changes here are the culprit for the trouble we're currently seeing in the easyconfigs test suite...

jfgrimm added a commit that referenced this pull request Nov 12, 2024
revert changes from PR #3472, seems to be leading to an infinite loop in easyconfigs test suite?
@@ -198,7 +199,7 @@ def __init__(self, *args, **kwargs):
if comp_cfg['patches']:
self.cfg.update('patches', comp_cfg['patches'])

self.comp_cfgs.append(comp_cfg)
self.comp_instances.append((comp_cfg, comp_cfg.easyblock(comp_cfg)))
Copy link
Contributor Author

@Thyre Thyre Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@boegel, @lexming This may be the issue causing infinite loop. Changing this to:

-self.comp_instances.append((comp_cfg, comp_cfg.easyblock(comp_cfg)))
+self.comp_instances.append((comp_cfg, comp_cfg.easyblock))

certainly improves the situation.
I'll continue checking if the test passes locally now.

Copy link
Contributor Author

@Thyre Thyre Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test suite is now stuck for QuantumESPRESSO. That's a huge progress (since we were previously stuck even for the first few EasyConfigs with A), but still not a complete fix.

Copy link
Contributor Author

@Thyre Thyre Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The QuantumESPRESSO check is not stuck. Instead, the logging files are not closed and all (?) tests after that are logged as well, causing around 11GB logs on my system. This also happens with develop as far as I can tell.

total 11G
drwx------  2 jreuter jreuter  800 12. Nov 20:44 .
drwxrwxrwt 24 root    root     540 12. Nov 20:36 ..
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-5.3.0-20241112.203827.pyJFM.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-5.4.0-20241112.203827.FyduX.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-5.4.0-20241112.203827.hzkuy.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.0-20241112.203827.FzleV.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.1-20241112.203827.FZgWV.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.2.1-20241112.203827.xMxaO.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.2-20241112.203827.fABEs.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.3-20241112.203827.kaPJh.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.3-20241112.203827.nZhTm.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.4.1-20241112.203827.juhTP.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.5-20241112.203827.pfDnY.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.5-20241112.203827.VBbnI.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.bRTCU.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.ewZNc.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.IzdWy.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.PgGZc.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.vwIEO.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.KUUtU.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.piJmd.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.rQAjF.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.sbJUP.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.ZBRpm.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.ZSOpP.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.8-20241112.203827.cKErR.log
-rw-r--r--  1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.8-20241112.203827.wwDpe.log
-rw-r--r--  1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.8-20241112.203827.XATBJ.log
-rw-r--r--  1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.0-20241112.203827.KuImR.log
-rw-r--r--  1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.0-20241112.203827.SXBiL.log
-rw-r--r--  1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.1-20241112.203827.BWcVK.log
-rw-r--r--  1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.1-20241112.203827.tpQDP.log
-rw-r--r--  1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.2-20241112.203827.GrtNk.log
-rw-r--r--  1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.2-20241112.203827.MArZe.log
-rw-r--r--  1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.2-20241112.203828.qrsQj.log
-rw-r--r--  1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3.1-20241112.203828.gMpKu.log
-rw-r--r--  1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3.1-20241112.203828.kpFyU.log
-rw-r--r--  1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3.1-20241112.203828.lrPYN.log
-rw-r--r--  1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3-20241112.203828.TOcVA.log
-rw-r--r--  1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3-20241112.203828.wnnQg.log

Could this be related to the way the EasyBlock is written?

@lexming
Copy link
Contributor

lexming commented Nov 12, 2024

@Thyre the changes in this PR are still alive in 5.0.x branch (https://github.com/easybuilders/easybuild-easyblocks/blob/5.0.x/easybuild/easyblocks/generic/bundle.py). We can focus on that branch for trobleshooting and fixing.

@Thyre
Copy link
Contributor Author

Thyre commented Nov 12, 2024

@Thyre the changes in this PR are still alive in 5.0.x branch (https://github.com/easybuilders/easybuild-easyblocks/blob/5.0.x/easybuild/easyblocks/generic/bundle.py). We can focus on that branch for trobleshooting and fixing.

@lexming We found the issue, see easybuilders/easybuild-easyconfigs#21841. It basically just boils down to loggers not being closed during the EasyConfig tests. I'll prepare an updated PR with a fix for this (and the comment regarding clang_aomp.py) tomorrow.

Thyre added a commit to Thyre/easybuild-easyblocks that referenced this pull request Nov 15, 2024
Thyre added a commit to Thyre/easybuild-easyblocks that referenced this pull request Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bundle easyblock ignores make_module_req_guess() from components
5 participants