Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement parallelization across broadcasted dimensions #376

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

astrofrog
Copy link
Member

@astrofrog astrofrog commented Jul 14, 2023

This builds on #374 to add a possible mode where if we are dealing with arrays with extra/broadcasted dimensions, we can split up the input array into chunks along the broadcasted dimensions which means we don't have to use the workaround of writing the whole dask array to a memmap.

For example, if we are reprojecting an array with dimensions (100, 2048, 2048) to (100, 1024, 1024), reprojecting only the celestial part of the WCS (the second and third dimensions of the array), we can specify e.g. block_size=(5, 1024, 1024) and this will cause the input to also be split up into chunks of (5, 2048, 2048) and each chunk of length 5 along the extra dimension will be handled separately.

One thing I'm not sure about is that at the moment we rely on the block size to fulfill certain conditions to toggle between the two main reprojection modes - but would it be cleaner to have a kwarg to do this?

@svank - your input would be appreciated too since you added the broadcasting functionality!

@codecov
Copy link

codecov bot commented Jul 14, 2023

Codecov Report

Merging #376 (b07342b) into main (eabeb23) will decrease coverage by 0.48%.
The diff coverage is 87.50%.

@@            Coverage Diff             @@
##             main     #376      +/-   ##
==========================================
- Coverage   93.60%   93.12%   -0.48%     
==========================================
  Files          25       25              
  Lines         891      902      +11     
==========================================
+ Hits          834      840       +6     
- Misses         57       62       +5     
Files Changed Coverage Δ
reproject/common.py 89.56% <87.50%> (-3.71%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@astrofrog astrofrog force-pushed the parallel-broadcasting branch from f999928 to 31ac87c Compare September 14, 2023 11:24
@astrofrog astrofrog marked this pull request as ready for review September 14, 2023 14:37
@astrofrog astrofrog requested a review from Cadair September 14, 2023 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant