Skip to content

Commit

Permalink
Solved 995.
Browse files Browse the repository at this point in the history 
Hello, I’ve prepared some code on my fork and re-based it with the latest upstream
, but I haven’t submitted the PR yet as my test environment isn’t fully ready. However, if you want to review the changes, you can check out my issue OSOceanAcoustics#995 branch. The commit is quite substantial since it touches multiple areas of the code, so I wanted to give you a heads-up on my approach.

For every instance where sonar_model was being checked against model lists for device-specific routines, I replaced it with a single constant command. This eliminates the need to modify the code in those places, as it now applies uniformly to all sonar models. However, device-specific functions still run on a per-device basis, as the SONAR_MODELS dictionary now includes new keys for each model. The values of these keys reference sonar-specific functions, where I’ve wrapped your existing code into functions. These functions live in the core.py script alongside the SONAR_MODELS. This follows the suggestion made by @emiliom.

For devices where no specific action is needed, I’ve implemented empty lambda functions that follow the same parameter schema.

I should mention that there are additional changes on my branch that originally aimed to address issue OSOceanAcoustics#995, but they’ve expanded beyond the initial scope. However, I believe this new feature is valuable enough to keep (though it’s still unfinished until I can correct it for other models beyond the EK60 and EK80, which are complete once they pass the checks).

The feature: I found a way to automatically parse the sonar_model from the raw data using open raw, by checking the data right at the outset. I’ve set the sonar_model argument to None by default, making it optional. For EK60 and EK80 raw data, if a sonar_model argument is provided, it’s ignored in favor of what’s available in the raw data. For other models, we still require the second argument, but only until I finish the parsing code for those models as well. I’ve mostly repurposed code that the devs already wrote.

As I mentioned, my testing environment is still incomplete, but once I have the necessary test data, I should be able to pass the checks.
  • Loading branch information
@spacetimeengineer
spacetimeengineer committed Oct 25, 2024
1 parent e938737 commit ec27512
Show file tree
Hide file tree
Showing 7 changed files with 236 additions and 112 deletions.
2 changes: 1 addition & 1 deletion .ci_helpers/docker/setup-services.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def parse_args():
parser.add_argument("--deploy", action="store_true", help="Flag to setup docker services")
parser.add_argument(
"--http-server",
default="docker_httpserver_1",
default="docker-httpserver-1",
help="Flag for specifying docker http server id.",
)
parser.add_argument(
Expand Down
38 changes: 15 additions & 23 deletions echopype/calibrate/api.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
from ..utils.prov import echopype_prov_attrs, source_files_vars
from .calibrate_azfp import CalibrateAZFP
from .calibrate_ek import CalibrateEK60, CalibrateEK80
from ..core import SONAR_MODELS

CALIBRATOR = {
"EK60": CalibrateEK60,
Expand All @@ -32,22 +33,11 @@ def _compute_cal(
waveform_mode = "BB" if waveform_mode == "FM" else waveform_mode

# Check on waveform_mode and encode_mode inputs
if echodata.sonar_model == "EK80":
if waveform_mode is None or encode_mode is None:
raise ValueError("waveform_mode and encode_mode must be specified for EK80 calibration")
check_input_args_combination(waveform_mode=waveform_mode, encode_mode=encode_mode)
elif echodata.sonar_model in ("EK60", "AZFP"):
if waveform_mode is not None and waveform_mode != "CW":
logger.warning(
"This sonar model transmits only narrowband signals (waveform_mode='CW'). "
"Calibration will be in CW mode",
)
if encode_mode is not None and encode_mode != "power":
logger.warning(
"This sonar model only record data as power or power/angle samples "
"(encode_mode='power'). Calibration will be done on the power samples.",
)


SONAR_MODELS[echodata.sonar_model]['waveform_mode_check'](waveform_mode)
SONAR_MODELS[echodata.sonar_model]['encode_mode_check'](encode_mode)


# Set up calibration object
cal_obj = CALIBRATOR[echodata.sonar_model](
echodata,
Expand Down Expand Up @@ -83,13 +73,15 @@ def add_attrs(cal_type, ds):
}[cal_type],
"units": "dB",
}
if echodata.sonar_model == "EK80":
ds[cal_type] = ds[cal_type].assign_attrs(
{
"waveform_mode": waveform_mode,
"encode_mode": encode_mode,
}
)






SONAR_MODELS[echodata.sonar_model]['add_attrs_check'](cal_type, ds, waveform_mode, encode_mode)



add_attrs(cal_type, cal_ds)

Expand Down
17 changes: 3 additions & 14 deletions echopype/calibrate/calibrate_base.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from ..echodata import EchoData
from ..utils.log import _init_logger
from .ecs import ECSParser
from ..core import SONAR_MODELS

logger = _init_logger(__name__)

Expand Down Expand Up @@ -98,20 +99,8 @@ def _check_echodata_backscatter_size(self):
If the size is above 2 GiB, raises a warning showing a recommended workflow
that will not overwhelm the system memory.
"""
# Initialize total nbytes
if self.echodata.sonar_model in ["EK60", "AZFP"]:
total_nbytes = self.echodata["Sonar/Beam_group1"]["backscatter_r"].nbytes
elif self.echodata.sonar_model == "EK80":
# Select source of backscatter data
beam = self.echodata[self.ed_beam_group]

# Go through waveform and encode cases
if (self.waveform_mode == "BB") or (
self.waveform_mode == "CW" and self.encode_mode == "complex"
):
total_nbytes = beam["backscatter_r"].nbytes + beam["backscatter_i"].nbytes
elif self.waveform_mode == "CW" and self.encode_mode == "power":
total_nbytes = beam["backscatter_r"].nbytes

total_nbytes = SONAR_MODELS[self.echodata.sonar_model]['backscatter_check'](self.echodata, self.ed_beam_group, self.waveform_mode, self.encode_mode)

# Compute GigaBytes from Bytes
total_gb = total_nbytes / (1024**3)
Expand Down
44 changes: 5 additions & 39 deletions echopype/calibrate/range.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,6 @@

DIMENSION_ORDER = ["channel", "ping_time", "range_sample"]

def range_meter_modification_handler(sonar_model):
return str(SONAR_MODELS[sonar_model]['parser']).split(".Parse")[1].split("'>")[0]

def compute_range_AZFP(echodata: EchoData, env_params: Dict, cal_type: str) -> xr.DataArray:
"""
Computes the range (``echo_range``) of AZFP backscatter data in meters.
Expand Down Expand Up @@ -149,19 +146,12 @@ def compute_range_EK(
or as power/angle combinations (``encode_mode="power"``) in a format
similar to those recorded by EK60 echosounders (the "power/angle" format).
"""
# sound_speed should exist already
#if echodata.sonar_model in ("EK60", "ES70"):
# ek_str = "EK60"
#elif echodata.sonar_model in ("EK80", "ES80", "EA640"):
# ek_str = "EK80"
#else:
# raise ValueError("The specified sonar_model is not supported!")

if "sound_speed" not in env_params:
raise RuntimeError(
"sounds_speed not included in env_params, "
f"use echopype.calibrate.env_params.get_env_params_{range_meter_modification_handler(echodata.sonar_model)}() to compute env_params "
)
"sounds_speed not included in env_params, " +
f"use echopype.calibrate.env_params.get_env_params_"+str(SONAR_MODELS[echodata.sonar_model]['parser']).split(".Parse")[1].split("'>")[0]+f"() to compute env_params "
)
else:
sound_speed = env_params["sound_speed"]

Expand All @@ -172,7 +162,7 @@ def compute_range_EK(
if chan_sel is None
else echodata[ed_beam_group].sel(channel=chan_sel)
)

# Range in meters, not modified for TVG compensation
range_meter = beam["range_sample"] * beam["sample_interval"] * sound_speed / 2
# make order of dims conform with the order of backscatter data
Expand Down Expand Up @@ -208,32 +198,8 @@ def range_mod_TVG_EK(
ref: https://github.com/CI-CMG/pyEcholab/blob/RHT-EK80-Svf/echolab2/instruments/EK80.py#L4297-L4308 # noqa
"""

def mod_Ex60():
# 2-sample shift in the beginning
return 2 * beam["sample_interval"] * sound_speed / 2 # [frequency x range_sample]

def mod_Ex80():
mod = sound_speed * beam["transmit_duration_nominal"] / 4
if isinstance(mod, xr.DataArray) and "time1" in mod.coords:
mod = mod.squeeze().drop_vars("time1")
return mod

beam = echodata[ed_beam_group]
vend = echodata["Vendor_specific"]

# If EK60
if echodata.sonar_model in ["EK60", "ES70"]:
range_meter = range_meter - mod_Ex60()

# If EK80:
# - compute range first assuming all channels have Ex80 style hardware
# - change range for channels with Ex60 style hardware (GPT)
elif echodata.sonar_model in ["EK80", "ES80", "EA640"]:
range_meter = range_meter - mod_Ex80()

# Change range for all channels with GPT
if "GPT" in vend["transceiver_type"]:
ch_GPT = vend["transceiver_type"] == "GPT"
range_meter.loc[dict(channel=ch_GPT)] = range_meter.sel(channel=ch_GPT) - mod_Ex60()

range_meter = SONAR_MODELS[echodata.sonar_model]['range_meter'](range_meter, beam, vend, sound_speed)
return range_meter
67 changes: 35 additions & 32 deletions echopype/convert/api.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -200,39 +200,42 @@ def _save_groups_to_file(echodata, output_path, engine, compress=True, **kwargs)
**kwargs,
)

SONAR_MODELS[echodata.sonar_model]['save_file_check'](echodata, output_path, engine, compress, COMPRESSION_SETTINGS, BEAM_SUBGROUP_DEFAULT, **kwargs)


# /Sonar/Beam_groupX group
if echodata.sonar_model == "AD2CP":
for i in range(1, len(echodata["Sonar"]["beam_group"]) + 1):
io.save_file(
echodata[f"Sonar/Beam_group{i}"],
path=output_path,
mode="a",
engine=engine,
group=f"Sonar/Beam_group{i}",
compression_settings=COMPRESSION_SETTINGS[engine] if compress else None,
**kwargs,
)
else:
io.save_file(
echodata[f"Sonar/{BEAM_SUBGROUP_DEFAULT}"],
path=output_path,
mode="a",
engine=engine,
group=f"Sonar/{BEAM_SUBGROUP_DEFAULT}",
compression_settings=COMPRESSION_SETTINGS[engine] if compress else None,
**kwargs,
)
if echodata["Sonar/Beam_group2"] is not None:
# some sonar model does not produce Sonar/Beam_group2
io.save_file(
echodata["Sonar/Beam_group2"],
path=output_path,
mode="a",
engine=engine,
group="Sonar/Beam_group2",
compression_settings=COMPRESSION_SETTINGS[engine] if compress else None,
**kwargs,
)
# if echodata.sonar_model == "AD2CP":
# for i in range(1, len(echodata["Sonar"]["beam_group"]) + 1):
# io.save_file(
# echodata[f"Sonar/Beam_group{i}"],
# path=output_path,
# mode="a",
# engine=engine,
# group=f"Sonar/Beam_group{i}",
# compression_settings=COMPRESSION_SETTINGS[engine] if compress else None,
# **kwargs,
# )
# else:
# io.save_file(
# echodata[f"Sonar/{BEAM_SUBGROUP_DEFAULT}"],
# path=output_path,
# mode="a",
# engine=engine,
# group=f"Sonar/{BEAM_SUBGROUP_DEFAULT}",
# compression_settings=COMPRESSION_SETTINGS[engine] if compress else None,
# **kwargs,
# )
# if echodata["Sonar/Beam_group2"] is not None:
# # some sonar model does not produce Sonar/Beam_group2
# io.save_file(
# echodata["Sonar/Beam_group2"],
# path=output_path,
# mode="a",
# engine=engine,
# group="Sonar/Beam_group2",
# compression_settings=COMPRESSION_SETTINGS[engine] if compress else None,
# **kwargs,
# )

# Vendor_specific group
io.save_file(
Expand Down
Loading

0 comments on commit ec27512

Please sign in to comment.