Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Map tests to software builds #673

Merged

Conversation

casparvl
Copy link
Collaborator

@casparvl casparvl commented Aug 19, 2024

The tests from the EESSI test suite that are run should depend on which software was built. This PR implements that functionality. Roughly speaking, it does the following:

  • Get the new modules from module_files.list.txt (in the tempdir)
  • Use yaml config file (software_to_test.yml) that maps regular expressions matching the (new) module names to a (list of) test(s) that need to be ran for that particular module name. First match in the yml file will be returned, so order is important.
  • A python code imports both the module_files.list.txt and software_to_test.yml, loops over all build software, each time executing the regular expression matches on the content of the software_to_test.yml, thus finding the tests that should be run for that particular module. If that test wasn't part of an aggregate list yet, it'll append it. Finally, the list is turned into a ReFrame argument string of the form e.g. -n EESSI_GROMACS -n EESSI_ESPRESSO, ... etc.
  • This argument list will be passed to the ReFrame command in the test step

@casparvl casparvl added the enhancement New feature or request label Aug 19, 2024
Copy link

eessi-bot bot commented Aug 19, 2024

Instance eessi-bot-mc-aws is configured to build for:

  • architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
  • repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

Copy link

eessi-bot bot commented Aug 19, 2024

Instance eessi-bot-mc-azure is configured to build for:

  • architectures: x86_64/amd/zen4
  • repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-software

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:zen3

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen3
  • handling command build repository:eessi.io-2023.06-software architecture:zen3 resulted in:

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen3
  • handling command build repository:eessi.io-2023.06-software architecture:zen3 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Aug 19, 2024

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.08/pr_673/16528

date job status comment
Aug 19 10:39:11 UTC 2024 submitted job id 16528 awaits release by job manager
Aug 19 10:39:54 UTC 2024 released job awaits launch by Slurm scheduler
Aug 19 10:40:55 UTC 2024 running job 16528 is running
Aug 19 10:57:12 UTC 2024 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-16528.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
No artefacts were created or found.
Aug 19 10:57:12 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 18/18 test case(s) from 18 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-16528.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

Hmmm... I guess dumping the module_file.list.txt in the current directory in the tarball step doesn't make it available in the test step. Maybe the working dirs are different. I'll try interactively to see what is going on.

ESC[32mSuccesfully ran 'reframe --version'ESC[0m
Loaded mappings from 'tests/eessi_test_mapping/software_to_tests.yml'
Traceback (most recent call last):
  File "/project/60006/SHARED/jobs/2024.08/pr_673/event_422aba90-5e17-11ef-9169-227fc6b2fccc/run_000/linux_x86_64_amd_zen3/eessi.io-2023.06-software/tests/eessi_test_mapping/map_software_to_test.py", line 78, in <module>
    main(args.mapping_file, args.module_list, args.debug)
  File "/project/60006/SHARED/jobs/2024.08/pr_673/event_422aba90-5e17-11ef-9169-227fc6b2fccc/run_000/linux_x86_64_amd_zen3/eessi.io-2023.06-software/tests/eessi_test_mapping/map_software_to_test.py", line 42, in main
    software_names = read_software_names(module_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/60006/SHARED/jobs/2024.08/pr_673/event_422aba90-5e17-11ef-9169-227fc6b2fccc/run_000/linux_x86_64_amd_zen3/eessi.io-2023.06-software/tests/eessi_test_mapping/map_software_to_test.py", line 17, in read_software_names
    raise FileNotFoundError(f"Error: {file_path} does not exist.")
FileNotFoundError: Error: module_file.list.txt does not exist.

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:zen3

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen3
  • handling command build repository:eessi.io-2023.06-software architecture:zen3 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen3 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen3
  • handling command build repository:eessi.io-2023.06-software architecture:zen3 resulted in:

Copy link

eessi-bot bot commented Aug 19, 2024

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.08/pr_673/16635

date job status comment
Aug 19 13:21:18 UTC 2024 submitted job id 16635 awaits release by job manager
Aug 19 13:21:26 UTC 2024 released job awaits launch by Slurm scheduler
Aug 19 13:27:29 UTC 2024 running job 16635 is running
Aug 19 13:42:46 UTC 2024 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-16635.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen3-1724074082.tar.gzsize: 0 MiB (132484 bytes)
entries: 24
modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all
patchelf/0.17.2-GCCcore-12.2.0.lua
software under 2023.06/software/linux/x86_64/amd/zen3/software
patchelf/0.17.2-GCCcore-12.2.0
other under 2023.06/software/linux/x86_64/amd/zen3
no other files in tarball
Aug 19 13:42:46 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 18/18 test case(s) from 18 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-16635.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:zen2

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Aug 19, 2024

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.08/pr_673/16636

date job status comment
Aug 19 13:40:28 UTC 2024 submitted job id 16636 awaits release by job manager
Aug 19 13:40:43 UTC 2024 released job awaits launch by Slurm scheduler
Aug 19 13:45:49 UTC 2024 running job 16636 is running
Aug 19 13:47:51 UTC 2024 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-16636.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-1724075235.tar.gzsize: 0 MiB (132315 bytes)
entries: 24
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
patchelf/0.17.2-GCCcore-12.2.0.lua
software under 2023.06/software/linux/x86_64/amd/zen2/software
patchelf/0.17.2-GCCcore-12.2.0
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Aug 19 13:47:51 UTC 2024 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-16636.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:zen2

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Aug 19, 2024

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.08/pr_673/16638

date job status comment
Aug 19 14:20:34 UTC 2024 submitted job id 16638 awaits release by job manager
Aug 19 14:21:03 UTC 2024 released job awaits launch by Slurm scheduler
Aug 19 14:22:05 UTC 2024 running job 16638 is running
Aug 19 14:25:09 UTC 2024 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-16638.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-1724077376.tar.gzsize: 0 MiB (132200 bytes)
entries: 24
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
patchelf/0.17.2-GCCcore-12.2.0.lua
software under 2023.06/software/linux/x86_64/amd/zen2/software
patchelf/0.17.2-GCCcore-12.2.0
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Aug 19 14:25:09 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 8/8 test case(s) from 8 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-16638.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

…sts are always added at the end, even if all test had other matches
@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:zen2

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Aug 19, 2024

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.08/pr_673/16639

date job status comment
Aug 19 14:32:56 UTC 2024 submitted job id 16639 awaits release by job manager
Aug 19 14:33:12 UTC 2024 released job awaits launch by Slurm scheduler
Aug 19 14:34:14 UTC 2024 running job 16639 is running
Aug 19 14:38:18 UTC 2024 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-16639.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-1724078094.tar.gzsize: 0 MiB (132141 bytes)
entries: 24
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
patchelf/0.17.2-GCCcore-12.2.0.lua
software under 2023.06/software/linux/x86_64/amd/zen2/software
patchelf/0.17.2-GCCcore-12.2.0
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Aug 19 14:38:18 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 8/8 test case(s) from 8 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-16639.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

This looks like it should be:

Loaded mappings from 'tests/eessi_test_mapping/software_to_tests.yml'
Read software names from 'module_files.list.txt'
Software: patchelf/0.17.2-GCCcore-12.2.0 -> Tests: ['EESSI_OSU']
Adding default set of tests: ['EESSI_OSU']
Full list of tests to run: ['EESSI_OSU']
Argument string: -n EESSI_OSU
-n EESSI_OSU
ESC[32mSuccesfully extracted names of tests to run: -n EESSI_OSUESC[0m
Listing tests: reframe --tag CI --tag 1_node --nocolor -n EESSI_OSU --list

Just for the sake of argument, let me retry this with a change to the mapping file so that EESSI_ESPRESSO test's are also run for patchelf.

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:zen2

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Aug 19, 2024

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.08/pr_673/16640

date job status comment
Aug 19 14:43:59 UTC 2024 submitted job id 16640 awaits release by job manager
Aug 19 14:44:21 UTC 2024 released job awaits launch by Slurm scheduler
Aug 19 14:45:24 UTC 2024 running job 16640 is running
Aug 19 14:58:44 UTC 2024 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-16640.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-1724078763.tar.gzsize: 0 MiB (132156 bytes)
entries: 24
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
patchelf/0.17.2-GCCcore-12.2.0.lua
software under 2023.06/software/linux/x86_64/amd/zen2/software
patchelf/0.17.2-GCCcore-12.2.0
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Aug 19 14:58:44 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 14/14 test case(s) from 14 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-16640.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

bot: build repo:eessi.io-2023.06-software arch:zen2

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-aws (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

Copy link

eessi-bot bot commented Aug 19, 2024

Updates by the bot instance eessi-bot-mc-azure (click for details)
  • received bot command build repo:eessi.io-2023.06-software arch:zen2 from casparvl

    • expanded format: build repository:eessi.io-2023.06-software architecture:zen2
  • handling command build repository:eessi.io-2023.06-software architecture:zen2 resulted in:

    • no jobs were submitted

Copy link

eessi-bot bot commented Aug 19, 2024

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.08/pr_673/16643

date job status comment
Aug 19 14:59:31 UTC 2024 submitted job id 16643 awaits release by job manager
Aug 19 14:59:47 UTC 2024 released job awaits launch by Slurm scheduler
Aug 19 15:00:49 UTC 2024 running job 16643 is running
Aug 19 15:14:37 UTC 2024 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-16643.out
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-1724079689.tar.gzsize: 0 MiB (130984 bytes)
entries: 24
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
patchelf/0.17.2-GCCcore-12.2.0.lua
software under 2023.06/software/linux/x86_64/amd/zen2/software
patchelf/0.17.2-GCCcore-12.2.0
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Aug 19 15:14:37 UTC 2024 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 14/14 test case(s) from 14 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-16643.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Collaborator Author

Ok, this looks exactly as it was intended:

Loaded mappings from 'tests/eessi_test_mapping/software_to_tests.yml'
Read software names from 'module_files.list.txt'
Software: patchelf/0.17.2-GCCcore-12.2.0 -> Tests: ['EESSI_ESPRESSO']
Adding default set of tests: ['EESSI_OSU']
Full list of tests to run: ['EESSI_ESPRESSO', 'EESSI_OSU']
Argument string: -n EESSI_ESPRESSO -n EESSI_OSU
ESC[32mSuccesfully extracted names of tests to run: -n EESSI_ESPRESSO -n EESSI_OSUESC[0m
Listing tests: reframe --tag CI --tag 1_node --nocolor -n EESSI_ESPRESSO -n EESSI_OSU --list

@casparvl casparvl marked this pull request as ready for review August 19, 2024 15:47
Copy link
Collaborator

@bedroge bedroge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, really nice improvement!

@bedroge bedroge merged commit f67d903 into EESSI:2023.06-software.eessi.io Sep 17, 2024
35 checks passed
@casparvl casparvl deleted the map_tests_to_software_builds branch September 18, 2024 18:57
OSU-Micro-Benchmarks/*:
- EESSI_OSU_Micro_Benchmarks
GROMACS/*:
- EESSI_GROMACS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casparvl We'll need to maintain this as we add more tests to the test suite, right?

Wouldn't it make more sense to make sure that we consistently use EESSI_<software_name>, so we don't need this mapping file at all?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants