DO NOT SQUASH Add PyOP2 to Firedrake repository #3817

JDBetteridge · 2024-10-22T15:34:13Z

Description

The big merge!

Closes #3877

Key changes introduced by this PR

PyOP2 is ingested into this repository (TSFC will be done later)
Existing tests are moved into a tests/firedrake subdirectory
Adds a pyproject.toml
pip install git+https://github.com/firedrakeproject/firedrake.git becomes a viable install method (subject to setting a number of environment variables). This ultimately will be our path to a Firedrake wheel.
Removes firedrake_configuration. The information is all accessible by inspecting PETSc variables.

Outstanding tasks

CI passing
Make sure that firedrake-install, firedrake-update, firedrake-status and firedrake-clean pass locally (these are not fully tested in the CI).
git rebase -i to try and keep a clean history.
Fix firedrake-zenodo as PyOP2 repo is going away.
Check manual for possible inaccuracies about install and Zenodo.

Following feedback

firedrake-update using an existing installation
firedrake-zenodo should be found

Future work

Also include TSFC in Firedrake repo (and put GEM and FInAT into FIAT).
pip install libsupermesh (nearly ready to go: JDBetteridge/pip install libsupermesh#10)

Make PETSc and petsc4py for CI use Firedrake not pip

actions: Build docs at end of successful test run

* Change requirements on loopy to point at the loopy sprint branch. * Replace lp.register_callable_kernel with lp.merge knl = lp.register_callable_kernel(knl, callee) is now knl = lp.merge([knl, callee]) . Reason given for loopy breaking back-compat: earlier they had a notion of a special "root_kernel" which only permitted one level of nesting loopy kernel calls and that was too restrictive. * Loopy callables: adapt to new API for registering. * Loopy Callables: register callables on a kernel not functions. * name is static property on SolveCallable and INVCallable * Loopy callables: register the callables rather than functions and remove the lookup functions. * Some progress * Loopy callables: Adapt API of with_types. * INVCallable has name "inverse" in loopy rep * Always pass a target when generating loopy kernels, otherwise we cannot merge them with wrapper kernels. * flake8 * Maybe generate call to inner kernel correctly Still doesn't work later. * Revert "Maybe generate call to inner kernel correctly" This reverts commit 009ca19f3fb133d53affeab53cd016c9b55620a9. * Revert "Revert "Maybe generate call to inner kernel correctly"" This reverts commit c85b60e6382b4762076b2744227fa5a6d903b830. * Remove dead code * rep2loopy: get name of callee kernel for matching the args. * Add missing code target * Maybe handle loopy argument access correctly dat.zero still doesn't work * Promote dats with shape (n,) to shape (n, 1) This enables sweeping indices to pick up a direct loop. * Set lang_version on all make_function calls * Fix disjoint arguments to C-string kernels * Resolve review comments * Rename function_args -> kernel_parameters * WIP Match args for all kernels in the callables table besides for LACallables? Why? * Only match args for callables that are CallableKernels. * Pin decorator Decorator version 5 changes the API in ways that break the argument type checking in PyOP2. We're therefore pinning on the version that works. * catch up with numpy type changes * Set within_inames_is_final on instructions * Squash numpy warning * Set strides on GlobalArgs * Caching: produce key for loop Programs. * Check if kernel is a loopy Program. * loopy.program.Program -> loopy.Program * Adapt to loopy interface changes. * Maintain loopy automatic reshaping (was in loopy) Everything in loopycompat.py file was in loopy/transform/callable.py but is removed in inducer/loopy#327. This commit maintains compatibility but should be phased out. * Fix loopy.program -> loopy.translation_unit * dont lint automatic reshaping (was in loopy) * Lint to pass flake8 * Fix loopy automatic reshaping (was in loopy) Changes were copied from an out of date branch by accident. * Fix type case error (recursively) This mirrors inducer/loopy#326 * ? Maybe fix the last dimension error mismatch. Not sure this will break things in other places. * Codegen: Don't merge indices of extent 1, so that the instructions are schedulable. This will fix e.g. the failure of tests/extrusion/test_steady_advection_2D_extr.py * Testing: Update loopy branch * Squashed commit of the following: commit 5a89bfc Author: Connor Ward <[email protected]> Date: Mon May 10 10:44:33 2021 +0100 Use Firedrake PETSc and petsc4py commit 948cb7d Author: David Ham <[email protected]> Date: Wed Apr 21 13:30:50 2021 +0100 Pin decorator Decorator version 5 changes the API in ways that break the argument type checking in PyOP2. We're therefore pinning on the version that works. commit 121c2a6 Author: David Ham <[email protected]> Date: Wed Apr 21 13:45:58 2021 +0100 catch up with numpy type changes * Revert "Squashed commit of the following:" This reverts commit bd1916142f883d6a4f0371ca6de2c344f0c7e78a. * Squashed commit of the following: commit 9a90c9f Merge: f1daf21 c8d7ff6 Author: Lawrence Mitchell <[email protected]> Date: Tue May 11 12:07:33 2021 +0100 Merge pull request #618 from OP2/wence/fix/actions-doc-build actions: Build docs at end of successful test run commit c8d7ff6 Author: Lawrence Mitchell <[email protected]> Date: Tue May 11 11:41:20 2021 +0100 actions: Build docs at end of successful test run commit f1daf21 Merge: d0cc348 5a89bfc Author: Lawrence Mitchell <[email protected]> Date: Mon May 10 20:32:24 2021 +0100 Merge pull request #617 from connorjward/test-ci Make PETSc and petsc4py for CI use Firedrake not pip commit 5a89bfc Author: Connor Ward <[email protected]> Date: Mon May 10 10:44:33 2021 +0100 Use Firedrake PETSc and petsc4py commit 948cb7d Author: David Ham <[email protected]> Date: Wed Apr 21 13:30:50 2021 +0100 Pin decorator Decorator version 5 changes the API in ways that break the argument type checking in PyOP2. We're therefore pinning on the version that works. commit 121c2a6 Author: David Ham <[email protected]> Date: Wed Apr 21 13:45:58 2021 +0100 catch up with numpy type changes * Lint. * Jenkins * Make import global. * Drop package branches. Co-authored-by: Sophia Vorderwuelbecke <[email protected]> Co-authored-by: Lawrence Mitchell <[email protected]> Co-authored-by: Connor Ward <[email protected]> Co-authored-by: David Ham <[email protected]> Co-authored-by: Kaushik Kulkarni <[email protected]>

Something like this will be necessary if we want general symmetry group stuff in the maps. It is also convenient if you want to stage in a permutation from the canonical FIAT order.

Instead of pattern matching on node names to determine whether nodes should be relabelled, just use a name generation scheme for everything. Now we have one less problem.

…versionning

Add PETSc decorator to compile

More loopy interface changes: arg_id_to_val -> arg_id_to_arg

…problem

* Fix progress bar erros for parallel executions * Test for a parallel execution

* Add petsc_raises fixture to catch PETSc callback exceptions

* Fix assembly of Real matrices

* Refactor test_demos, avoid mpiexec invocation * Fix multigrid test, __main__ is not valid when importing

* Get value_shape from FunctionSpace * Define FunctionSpace.block_size as the number of dofs per node * sort_domains --------- Co-authored-by: ksagiyam <[email protected]>

* DO NOT MERGE * Test VOM with point variant * Update .github/workflows/build.yml * Test Regge point variant * add continuity test * Test H(curl div) element (#3843) * Test H(curl div) element * test Hu-Zhang * checkout ufl master branch * checkout master branches

Signed-off-by: Umberto Zerbinati <[email protected]> Co-authored-by: Umberto Zerbinati <[email protected]>

* Disk checkpointing managing start, pause and continue checkpointing. * AdjointDiskCheckpoint --------- Co-authored-by: David A. Ham <[email protected]>

* Moves PyOP2 code into Firedrake repository. * Adds a pyproject.toml for Firedrake * Enables (with some caveats) one to `pip install firedrake`! * Adds a new pip workflow to make sure things work * Moves Firedrake tests into a `tests/firedrake` subdirectory.

…pyop2_tsfc

ksagiyam · 2024-11-25T15:33:50Z

I did a clean install using the branch. Wanted to test firedrake-zenodo, but I got the following:

>>> firedrake-zenodo -t "My paper title"
firedrake-zenodo: command not found

firedrake-update seems almost working, but I had to copy firedrake/scripts/firedrake-install to scripts/firedrake-install.

firedrake/scripts/firedrake_clean.py

connorjward · 2024-11-25T15:51:45Z

I did a clean install using the branch. Wanted to test firedrake-zenodo, but I got the following:
>>> firedrake-zenodo -t "My paper title"
firedrake-zenodo: command not found

Thanks for noticing this. Will fix ASAP.

firedrake-update seems almost working, but I had to copy firedrake/scripts/firedrake-install to scripts/firedrake-install.

Specifically what failed? I thought I tried this and things worked.

ksagiyam · 2024-11-25T16:07:04Z

(firedrake) ksagiyam@ma-ksagiyam-2:~/cpython/test/firedrake/src/firedrake$ firedrake-update 
/home/ksagiyam/cpython/test/firedrake/bin/firedrake-update:20: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources.extern.packaging.version import Version, InvalidVersion
Running /home/ksagiyam/cpython/test/firedrake/bin/firedrake-update
BLAS config is {}
Configuration saved to /home/ksagiyam/cpython/test/firedrake/.configuration.json
Updating the git repository for firedrake
Creating firedrake-update script.
Traceback (most recent call last):
  File "/home/ksagiyam/cpython/test/firedrake/bin/firedrake-update", line 1902, in <module>
    build_update_script()
  File "/home/ksagiyam/cpython/test/firedrake/bin/firedrake-update", line 1407, in build_update_script
    with open("firedrake/scripts/firedrake-install", "r") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'firedrake/scripts/firedrake-install'

connorjward · 2024-11-25T16:10:13Z

(firedrake) ksagiyam@ma-ksagiyam-2:~/cpython/test/firedrake/src/firedrake$ firedrake-update 
/home/ksagiyam/cpython/test/firedrake/bin/firedrake-update:20: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources.extern.packaging.version import Version, InvalidVersion
Running /home/ksagiyam/cpython/test/firedrake/bin/firedrake-update
BLAS config is {}
Configuration saved to /home/ksagiyam/cpython/test/firedrake/.configuration.json
Updating the git repository for firedrake
Creating firedrake-update script.
Traceback (most recent call last):
  File "/home/ksagiyam/cpython/test/firedrake/bin/firedrake-update", line 1902, in <module>
    build_update_script()
  File "/home/ksagiyam/cpython/test/firedrake/bin/firedrake-update", line 1407, in build_update_script
    with open("firedrake/scripts/firedrake-install", "r") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'firedrake/scripts/firedrake-install'

But that's not what I have... https://github.com/firedrakeproject/firedrake/pull/3817/files#diff-e86a30193cf50f52a86c9839c05da17a3f06c2ff5d082571efc52644f7f79771L1407-R1400

Strange. I think that running firedrake-update is pulling the current master version of the script.

ksagiyam · 2024-11-25T16:22:56Z

To be clear, I installed current master, checked out this PR branch, and then ran firedrake-update. I think in this case bin/firedrake-update tries to pull from scripts/firedrake-update (

firedrake/scripts/firedrake-install

Line 1407 in b783253

with open("firedrake/scripts/firedrake-install", "r") as f:

), but it does not exist in the new firedrake structure.

connorjward · 2024-11-25T16:26:32Z

To be clear, I installed current master, checked out this PR branch, and then ran firedrake-update. I think in this case bin/firedrake-update tries to pull from scripts/firedrake-update (

firedrake/scripts/firedrake-install

Line 1407 in b783253

with open("firedrake/scripts/firedrake-install", "r") as f:

), but it does not exist in the new firedrake structure.

Ah I see. This is a case I had not (and certainly should have) considered.

connorjward and others added 30 commits May 10, 2021 17:21

Use Firedrake PETSc and petsc4py

5a89bfc

Merge pull request #617 from connorjward/test-ci

f1daf21

Make PETSc and petsc4py for CI use Firedrake not pip

actions: Build docs at end of successful test run

c8d7ff6

Merge pull request #618 from OP2/wence/fix/actions-doc-build

9a90c9f

actions: Build docs at end of successful test run

add standard set operations

afe4ad4

codegen/api: New PermutedMap class corresponding to map o permutation

ded8578

Something like this will be necessary if we want general symmetry group stuff in the maps. It is also convenient if you want to stage in a permutation from the canonical FIAT order.

codegen: No more regular expressions in node renaming

dfeec40

Instead of pattern matching on node names to determine whether nodes should be relabelled, just use a name generation scheme for everything. Now we have one less problem.

tests: Squash numpy deprecation warnings

a257d1d

Merge remote-tracking branch 'origin/master' into DataCarrier-object-…

07a47b3

…versionning

Add PETSc decorator to compile

1e74cea

add standard set operations

2781213

Merge branch 'fs_filter' of github.com:OP2/PyOP2 into fs_filter

855ec95

Add some missing DataCarrier inits

29973b1

Merge pull request #623 from OP2/connorjward/time-compilation

c45a751

Add PETSc decorator to compile

Merge remote-tracking branch 'origin/master' into fs_filter

bb27936

More loopy interface changes: arg_id_to_val -> arg_id_to_arg

e56d26f

Add recreate to Arg

f057908

Recreate method of Arg

3442f06

Fix cache problem

92184de

Fix cache problem

e9b73ab

Fix cache problem

f24478a

Requirements: update loopy fork.

b28bf14

Merge pull request #627 from OP2/new-loopy-master

781201c

More loopy interface changes: arg_id_to_val -> arg_id_to_arg

Merge remote-tracking branch 'origin/master' into Melina97/fix_cache_…

760dbe7

…problem

Updated changes

899a543

Don't use a parloop when not needed

70c4b39

added subset case for copy

58f89ef

@dham

2460b37

Don't use a parloop when not needed

3ddc70a

Ig-dolci and others added 12 commits November 25, 2024 10:49

Fix progress bar error for parallel executions (#3859)

4a7e745

* Fix progress bar erros for parallel executions * Test for a parallel execution

Fixes for updated PETSc (#3853)

c9d3922

* Add petsc_raises fixture to catch PETSc callback exceptions

function space: fix collapse (#3861)

654bb27

Fix assembly of Real matrices (#3846)

6fe96de

* Fix assembly of Real matrices

ngsPETSc "stable" release is coming ! (#3857)

bb8f6e1

Parallel test improvements (#3865)

dff63f8

* Refactor test_demos, avoid mpiexec invocation * Fix multigrid test, __main__ is not valid when importing

Get value_shape from FunctionSpace (#3862)

cf46e77

* Get value_shape from FunctionSpace * Define FunctionSpace.block_size as the number of dofs per node * sort_domains --------- Co-authored-by: ksagiyam <[email protected]>

Actually install ngsPETSc using pip (#3873)

f222dc5

Signed-off-by: Umberto Zerbinati <[email protected]> Co-authored-by: Umberto Zerbinati <[email protected]>

Disk checkpointing (#3812)

8f45d26

* Disk checkpointing managing start, pause and continue checkpointing. * AdjointDiskCheckpoint --------- Co-authored-by: David A. Ham <[email protected]>

test: add test for 0**{min_value, max_value} (#3855)

6bb9a9f

connorjward force-pushed the JDBetteridge/merge_pyop2_tsfc branch from 463eb23 to 1bceac4 Compare November 25, 2024 10:55

Merge remote-tracking branch 'origin/master' into JDBetteridge/merge_…

ccec788

…pyop2_tsfc

connorjward mentioned this pull request Nov 25, 2024

Revert "Hack dataclass'd function_interface to avoid breaking Firedrake" inducer/loopy#886

Merged

Use correct link to get firedrake-install script

c72810a

ksagiyam reviewed Nov 25, 2024

View reviewed changes

firedrake/scripts/firedrake_clean.py Outdated Show resolved Hide resolved

Fix firedrake-zenodo and firedrake-clean

8bcb96f

connorjward force-pushed the JDBetteridge/merge_pyop2_tsfc branch from 01273f3 to 8bcb96f Compare November 25, 2024 16:23

connorjward added 2 commits November 25, 2024 16:34

Revert firedrake-install location, to avoid breaking firedrake-update

867337f

Do not specify numpy>=2 and tweak install instructions

74a3f85

dham merged commit dab79d3 into master Nov 27, 2024
18 checks passed

dham deleted the JDBetteridge/merge_pyop2_tsfc branch November 27, 2024 16:10

vojtamolda mentioned this pull request Dec 5, 2024

Compatibility fix for Firedrake 2024-12 Docker image IvanYashchuk/firedrake-ts#24

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DO NOT SQUASH Add PyOP2 to Firedrake repository #3817

DO NOT SQUASH Add PyOP2 to Firedrake repository #3817

JDBetteridge commented Oct 22, 2024 •

edited by connorjward

Loading

ksagiyam commented Nov 25, 2024

connorjward commented Nov 25, 2024

ksagiyam commented Nov 25, 2024

connorjward commented Nov 25, 2024

ksagiyam commented Nov 25, 2024

connorjward commented Nov 25, 2024

DO NOT SQUASH Add PyOP2 to Firedrake repository #3817

DO NOT SQUASH Add PyOP2 to Firedrake repository #3817

Conversation

JDBetteridge commented Oct 22, 2024 • edited by connorjward Loading

Description

Key changes introduced by this PR

Outstanding tasks

Following feedback

Future work

ksagiyam commented Nov 25, 2024

connorjward commented Nov 25, 2024

ksagiyam commented Nov 25, 2024

connorjward commented Nov 25, 2024

ksagiyam commented Nov 25, 2024

connorjward commented Nov 25, 2024

JDBetteridge commented Oct 22, 2024 •

edited by connorjward

Loading