v.0.4.1

Overview

This release adds important functions (e.g., NASC data ingestion, semivariogram fitting) and substantial documentation expansions, along with a few bug fixes.

What's Changed

New features, bug fixes, and enhancements ✨

Software 🖥️

• Raw Echoview export ingestion and batch processing by @brandynlucca in #257, #274
  • This deprecates the need for the haul-to-transect key *.xlsx files
• New semivariogram fitting and optimization methods by @brandynlucca in #251
  • This includes a notebook GUI by @brandynlucca in #258, #267
• Adjustments to the kriging mesh cropping method by @brandynlucca in #269, #273
• Addressed a bug preventing INPFC from being used instead of KS for transect_analysis by @brandynlucca in #272
• Upgrades to summary console printouts of analysis results @brandynlucca in #255

Documentation 📖

• Differences between EchoPro and Echopop by @brandynlucca in #277
• Updates to the example workbook in the software documentation which reflect updated methods by @brandynlucca in #258, #267
• Updates to the description of the dataset columns in 'Input Files' by @brandynlucca in #283
• Description of how data files are loaded and preprocessed within Echopop by @brandynlucca in #274
• Theoretical description of how kriged biomass results can be converted into abundance and NASC by @brandynlucca, @leewujung in #216
• Improvements to the equations in both the kriged abundance and kriged biomass apportionment document pages by @leewujung in #280

Infrastructure 🏗️

• New internal checks for validating configuration and data file inputs have been added by @brandynlucca in #248, #271, #275
• Methods for translating various column names into those required in Echopop by @brandynlucca in #268
• Additional testing methods via tox by @brandynlucca in #251

Full Changelog: v0.4.0...v.0.4.1

v0.4.0

Overview

This is a major release that overhauls and renames the entire package. The changes include many bug fixes from the previous implementation, and a new set of documentation that continue to be built in subsequent releases.

What's new

New features, bug fixes, and enhancements

• Overhauled codebase with a new underlying data structure to Survey-class objects by @brandynlucca in #241
  • See the updated documentation for more information on the updated data structure and example workflow
• Overhauled data loader and core API that defines the expected data structure with built-in validators by @brandynlucca in #161
• Overhauled the kriging algorithm and biomass conversions by @brandynlucca in #206
• Added new population-level calculations, stratified statistics, and other metrics by @brandynlucca in #176
• New initialization and user-specific configuration yaml files to specify data files by @emiliom in #121

Infrastruture

• Added pytest tests by @leewujung in #174
  • Configured tests to run on python 3.9, 3.10, 3.11 by @leewujung in #231
• Renamed package from EchoPro to Echopop by @leewujung in #200
• Consolidated unused files, modules, notebooks, tests by @brandynlucca in #227
• pre-commit formatting implemented by @leewujung in #232
• Added Read the Docs documentation auto build by @leewujung in #182

New Contributors

• @brandynlucca made their first contribution in #161
• @pre-commit-ci made their first contribution in #238

Full Changelog: v0.3.1...v0.4.0

v0.3.1

Overview

The major new feature in this release is the generation of "report" Excel files, the primary quantitative export from EchoPro. A new notebook was added to illustrate the report generation workflow. In addition, the transect subset selection scheme was upgraded to reflect the Matlab implementation (but see #95); the test infrastructure was improved, including new tests; and the documentation was enhanced with the addition of auto-generated API documentation in the Jupyter Book and an update to the repository README file.

The Python EchoPro GitHub repository was made public in April. It was previously private while initial development was taking place.

New computations and functionality

• Python EchoPro can now generate the Excel file "reports" found in Matlab EchoPro (#18, PR #84). 15 reports are generated. This development entailed significant new computations to generate binned results before exporting to files, plus other computations.
• The transect subset selection scheme was updated from the previous interim scheme, to reflect the one found in Matlab EchoPro (#33, PR #89). But see #95 for a remaining issue.

Tests, packaging and documentation

• New test modules, for report generation and transect subset selection (PR's #84, #89, #99).
• Generalized and simplified most of the test path generation functionality in conftest.py and the test modules, with better handling of base paths, including a new HERE path accessible from _testing.py. One hard-wired path still remains, to be addresses in the next release (PR's #94, #84).
• Updated installation requirements, and conda environment file after testing with the latest dependencies. Included a couple of new version pinnings that will be addressed in a future release (PR #98).
• Updated installation instructions and other information in README.md (PR #98).
• New reports_workflow.ipynb Jupyter notebook (PR's #84).
• Sanitized input file names (in Google Drive and in survey_year_2019_config.yml), replacing & with _ to avoid issue of Google Drive downloads automatically renaming file names during zip-and-download of input files (#96, PR #97).
• Added auto-generated API documentation in the Jupyter Book (PR #90).

v0.3.0

Overview

This release includes major additions of functionality that largely complete the expected computational capabilities of Python EchoPro. Some of these additions necessitated changes in the API, such as creation and renaming of functions. Additional improvements include integration tests, packaging refinements and documentation.

New computations and functionality

• Comprehensive generation of biomass, biomass density, abundance and numerical density variables, with these variables being defined for the male and female populations over transect ("unkriged") and kriging mesh points. This work included the addition of abundance and numerical density calculations, and a harmonization of variable names related to those four core variables. (#61, #70, PR #60, PR #64, PR #67)
• Generation of binned distributions of those four core variables by age and length (and sex), including distribution over transect ("unkriged") and kriging mesh points. (#63, #77, PR #74, PR #78)
• API Changes.
  • The API call survey.compute_biomass_density() was renamed to survey.compute_transect_results()
  • Column names in some DataFrame and GeoDataFrame results were modified.
  • Several new API calls that produce DataFrame and GeoDataFrame results were added. See the Python EchoPro Workflow notebook for a complete demonstration.

Tests, packaging and documentation

• A new testing infrastructure was added, with three sets of tests: Tests for biomass, biomass density, abundance and numerical density (by sex) over transect points; similar tests over kriging mesh points; and tests for length-age-sex distributions over both transects and kriging mesh points. (PR #67, PR #70, PR #74, PR #80)
• The packaging setup was overhauled to ensure error-free installation in a user mode, in addition to development mode. A Python wheel file is being made available for such usage, and dependency version pinning was made less strict (#75, PR #68, PR #69, PR #76)
• Documentation was improved, particularly with the addition of a new Jupyter Book (https://uw-echospace.github.io/EchoPro/) hosting executed and rendered versions of the example notebooks and an overhaul of the README.md file (#75, PR #65, PR #76)

v0.2.0

Overview

This release is focused on three key improvements:

1. Revision of input data handling, particularly the names of columns in the data files and in corresponding data frames.
2. API refinements, especially a reorganization of modules into a couple of subpackages (folders) and root-level modules; and harmonization of a core set of variable names corresponding to EchoPro results.
3. Improvements to code consistency and error checking.

All notebooks were updated to reflect these changes.

No new functionality per se was added in this release.

Input data handling

• Previous decisions on column names in input data files were revisited and implemented in both the 2019 sample input data files and the code base. Columns (variables) not being used in EchoPro computations are now excluded more clearly from the results (#19, #44, PR #43, PR #47)
• A new input data file containing only the haul-to-transect mapping was introduced. It's intended to replace the use of the mapping found in the gear file (#45, PR #46)

API refinements and variable harmonization

• Reorganized modules into root-level modules, subpackages (folders) computation and data_loader, and utilities in utils. Includes breaking up some modules into more logical groups based on functionality (#48, #14, PR #54)
• Harmonized names associated with kriging and associated GeoDataFrames that hold final results (#52, PR #57)
• Harmonized key variable and function names involved in the computation of biomass density (#34, PR #51)
• Added arguments to map plotting functions to provide option to produce lighter-weight (fewer points) plots (#49, PR #53)

Code consistency and error checking.

• Added pre-commit automatic code checking and style enforcement bots on GitHub repository (PR #58)
• Improved scheme for input dataframe column checks and associated error output messsages (#28, PR #36)
• Extended cross-platform capability by allowing input configuration file paths to be pathlib Paths, converting input string paths to pathlib Paths, and usage of pathlib Paths for all input data paths created at runtime (#37, PR #36)

v0.1.0-alpha

Overview

This is the first pre-release of Python EchoPro. This release marks the completion of all core routines contained within the Matlab version of EchoPro and construction of the initial API. Specifically, the following core items have been completed:

• Creation of easily viewable DataFrames containing data related to the biological, NASC, and stratification files
• Computation of the normalized biomass density using the biological files
• Implementation of the Jolly-Hampton algorithm for both Kriged data and data without Kriging
• Routines to construct, transform, subselect, and plot the mesh associated with Kriging and the semi-variogram calculation
• Computation of the normalized semi-variogram and fitting of the semi-variogram model using ipywidgets
• A routine to perform Ordinary Kriging of the calculated biomass density over a provided mesh and plotting functions for the results
• The ability to select a subset of transects and perform any of the above core routines
• Implementation of bootstrapping (for Kriged data and data without Kriging) based on a reduction percentage of the total number of transects

2019 input files and package installation

Development and testing so far has been based on input files and corresponding Matlab EchoPro runs from the 2019 survey. The README.md file contains a link to download (if granted permission) the 2019 input files that can be used with this release, as well as installation instructions.