Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added sonar model checkers for EK60, EK80, ER60, AZFP6, AZFP, AD2CP and tests #1399

Open
wants to merge 31 commits into
base: main
Choose a base branch
from

Conversation

spacetimeengineer
Copy link
Contributor

Future releases of echopype may accommodate cases where the user is not required to provide sonar model numbers. We have here a simple checker to do that.

-Added model checkers for EK60 and EK80 sonar models.

spacetimeengineer and others added 3 commits October 25, 2024 15:33
Future releases of echopype may accommodate cases where the user is not required to provide sonar model numbers. We have here a simple checker to do that.

-Added model checkers for EK60 and EK80 sonar models.
Copy link
Contributor Author

@spacetimeengineer spacetimeengineer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't seem appropriate to put this function in the class, at least not yet but the file did seem a good place to put this.

@codecov-commenter
Copy link

codecov-commenter commented Oct 25, 2024

Codecov Report

Attention: Patch coverage is 92.59259% with 4 lines in your changes missing coverage. Please review.

Project coverage is 80.40%. Comparing base (9f56124) to head (80b6656).
Report is 152 commits behind head on main.

Files with missing lines Patch % Lines
echopype/convert/parse_azfp.py 85.71% 2 Missing ⚠️
echopype/convert/parse_ad2cp.py 80.00% 1 Missing ⚠️
echopype/convert/parse_azfp6.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1399      +/-   ##
==========================================
- Coverage   83.52%   80.40%   -3.12%     
==========================================
  Files          64       18      -46     
  Lines        5686     3138    -2548     
==========================================
- Hits         4749     2523    -2226     
+ Misses        937      615     -322     
Flag Coverage Δ
unittests 80.40% <92.59%> (-3.12%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@spacetimeengineer
Copy link
Contributor Author

I do not have coverage because I haven't wrote the tests yet. Ill be sure to get those over ASAP.

spacetimeengineer and others added 8 commits October 28, 2024 10:19
Added test cases for EK60 and EK80 raw parser functions to validate their identification of echosounder types through raw data alone.
The test failed because a proper path wasn't provided to them. This commit should fix that.
I forgot to import the outside-of-class functions in the __init__. Consider this fixed.
- Added notes about the potential need to remove sudo privileges for some users.
- Improved tests by adding an exception handler for key errors, returning False when encountered.
Syntax error fixed.
The 'e' variable commonly used in error handling was not utilized, so I removed it. This change caused tests to fail, prompting its removal.
…#1400)

Bumps [actions/cache](https://github.com/actions/cache) from 4.1.1 to 4.1.2.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4.1.1...v4.1.2)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…oustics#1401)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v5.2.0...v5.3.0)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@leewujung leewujung self-requested a review October 30, 2024 20:50
@leewujung
Copy link
Member

@spacetimeengineer: is this PR ready to review? If not just ping me when you think it's ready. Thanks!

@spacetimeengineer
Copy link
Contributor Author

spacetimeengineer commented Oct 30, 2024 via email

@spacetimeengineer
Copy link
Contributor Author

spacetimeengineer commented Oct 30, 2024 via email

@spacetimeengineer
Copy link
Contributor Author

On my fork I have having these errors pop up and I don't fully understand them yet.

minioci-build
Username and password required
http-build
The job was canceled because "minioci" failed.
http-build
The operation was canceled.

@leewujung
Copy link
Member

Oh I see - you can disable these builds since they will be run on the main repo (here).

@spacetimeengineer
Copy link
Contributor Author

Thanks, that’s great! Let me know if there's anything else I need to do. Just a heads-up—this PR is part of a series of similar checkers for other models. I don’t think this approach will be very effective until all models are covered, and even then, I’d recommend keeping the sonar_model parameter as an option at least. Let me know if this seems a worthwhile feature.

@spacetimeengineer
Copy link
Contributor Author

I wanted to provide some context: I decided to merge a few of my branches since the code was relevant enough. I just added more sonar_model checkers and tests.

@spacetimeengineer spacetimeengineer changed the title Added sonar model checkers for EK60, EK80 Added sonar model checkers for EK60, EK80, ER60, AZFP6, AZFP, AD2CP Nov 1, 2024
@spacetimeengineer spacetimeengineer changed the title Added sonar model checkers for EK60, EK80, ER60, AZFP6, AZFP, AD2CP Added sonar model checkers for EK60, EK80, ER60, AZFP6, AZFP, AD2CP and tests Nov 1, 2024
@spacetimeengineer spacetimeengineer self-assigned this Nov 1, 2024
@spacetimeengineer spacetimeengineer added the enhancement This makes echopype better label Nov 1, 2024
@leewujung
Copy link
Member

I haven't had a chance to check the code, but from a quick glance this is fine since it is not a gigantic PR.

I think it would be better to always keep the sonar_model argument, and spit out an error if what they specified mismatches with what are contained in the files (i.e. I don't think we should "make it work" under the hood for users if they specify the wrong parameters). It is part of making sure people are aware of what operations they are trying to perform, since some people may be new to echosounder processing.

@spacetimeengineer
Copy link
Contributor Author

spacetimeengineer commented Nov 1, 2024

I agree with your perspective. My intention was not to remove the sonar model parameter, (it only helps echopype users to provide more information) but rather to explore the possibility of allowing future releases to parse the sonar model directly from the raw data. This approach could enhance the intelligence of the base parser. While it wouldn't be fully effective until all model checkers are developed, it would allow us to clarify what the sonar model is not, rather than just asserting what it is. I figured if the user makes a mistake providing the sonar_model then the more intelligent base parser could provide a warning and ask the user if they want to process with the true parsed sonar_model value or some kind of override.

Need to import these functions.
Comment on lines -84 to +100
Docker image on Docker hub.
Docker image on Docker hub. There’s no need to pull directly from Docker Hub, as the scripts below will handle that for you. For Linux Users: If you run into an issue where accessing the Docker daemon requires sudo privileges, which may not suit your environment, you can adjust your settings to allow non-root access to Docker:

```shell
sudo groupadd docker
```

Add the current user to the Docker group
```shell
sudo gpasswd -a $USER docker
```

Restart the docker daemon
```shell
sudo service docker restart
```

You might also need to restart your computer if the steps above don't resolve the issue.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you move this to a new PR since it's under a completely different topic? I think there we could update the contributing guide from you and others as you're fresh from setting up a dev environment and may want to add other details. Thanks!

Comment on lines 38 to 42
with RawSimradFile(raw_file, "r", storage_options=storage_options) as fid:
config_datagram = fid.read(1)
config_datagram["timestamp"] = np.datetime64(
config_datagram["timestamp"].replace(tzinfo=None), "[ns]"
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure why you are accessing the timestamp here.

See Gavin's #494 (comment) that I think would be useful here to check the format without using the full blown parser. Implementing that into the checker would address #494 too in this PR.

Copy link
Member

@leewujung leewujung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spacetimeengineer : Thanks for the PR! I added some inline comments above, and have a few structural comments below:


I think the sonar model checking code should be in a separate module (say convert/sonar_model_checker.py), specified in core.py like the parser and set_group functions, and called in open_raw to check the matches between user-specified sonar_model type and the actual file content. The checks can happen right after the _check_file call that checks file existence and before the parser is called to parse the full data file. This way the functions will be easier to maintain since they don't get wrapped in the parser child classes but with different method names.

Similarly, I think the tests for these checker functions can be in a different module tests/convert/test_sonar_model_checker.py, to keep these tests isolated from the actual parsing of data file content.


The is_{SONAR_MODEL} functions you added are of 2 types:

  1. One checks for just the file extension name: is_AD2CP, is_AZFP6 -- this check already exists in functions validate_ext and validate_azfp_ext in core.py.
  2. One takes a peek "under the hood" to see if the data actually fits the expected config format: is_AZFP, is_EK60, is_ER60, is_EK80 -- we need these. Note that the older Simrad format was also used for many other echosounder models, not just EK60 and ER60. The same for the newer Simrad format. We can improve what's in core.py so that the same blocks using the same parser/setgrouper don't need to be repeated. I'll write a small issue for that -- we just need a mechanism to select the right key in the main SONAR_MODELS dict. -- in fact, this is just sonar_model handling, grouping #995.

For paths to test files, please use fixtures:

@pytest.fixture(scope="session")
def test_path():

This will make maintenance and upcoming migration of the testing infrastructure smoother.

@spacetimeengineer
Copy link
Contributor Author

@leewujung Acknowledged. Ill begin putting in these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This makes echopype better
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants