Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Frame extraction with reencoding #78

Merged
merged 37 commits into from
Nov 22, 2023
Merged

Conversation

sfmig
Copy link
Collaborator

@sfmig sfmig commented Nov 13, 2023

Some improvements / additions to the frame extraction slurm script

  • optionally reencode videos before extracting frames
  • save SLURM logs along with the extracted frames
  • save SLURM logs along with the reencoded videos
  • derive the file extension from the input video

I deleted the frame extraction only script and the script to extract frames locally.

This PR is a first step towards a more general bash script that would make use of the CLI tools and that we wouldn't edit for each run - some thoughts on that in this draft PR #88 (see this file specifically)

@codecov-commenter
Copy link

codecov-commenter commented Nov 20, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (224b2bd) 29.81% compared to head (53bbd8c) 29.81%.
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #78   +/-   ##
=======================================
  Coverage   29.81%   29.81%           
=======================================
  Files           6        6           
  Lines         332      332           
=======================================
  Hits           99       99           
  Misses        233      233           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sfmig sfmig marked this pull request as ready for review November 20, 2023 16:12
@sfmig sfmig requested a review from samcunliffe November 20, 2023 16:13
Copy link
Member

@samcunliffe samcunliffe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's been a while since I've written a longish bash script.

Even though it's a bit out of scope for this PR, we should think about how to test these guys. Like could we call them in CI with a --dryrun flag or something?

You might want to ignore my comments 👇 if there are plans in #88 (I didn't dig into that yet because: rebase.)

bash_scripts/run_reencode_and_frame_extraction_array.sh Outdated Show resolved Hide resolved
crabs/bboxes_labelling/extract_frames_to_label_w_sleap.py Outdated Show resolved Hide resolved
@sfmig
Copy link
Collaborator Author

sfmig commented Nov 22, 2023

It's been a while since I've written a longish bash script.

Even though it's a bit out of scope for this PR, we should think about how to test these guys. Like could we call them in CI with a --dryrun flag or something?

You might want to ignore my comments 👇 if there are plans in #88 (I didn't dig into that yet because: rebase.)

Yes, testing this would be ideal. I had a quick look and bats seems like a very nice tool for this. IIUC, it can be run in CI too.

Right now we manually edit this script for the different frame extraction jobs. So testing this would be cumbersome. My idea with #88 is to have a bash script that we keep intact, and to which we pass any required parameters as command line arguments. Specifically, we would pass a config.json file that we would read and print to the logs. This way we keep track of what input parameters we passed but the bash script is not edited every time.

So for now I would suggest to merge this PR and add these comments to PR #88. There are other options for unit testing bash scripts, I am adding a brief overview to PR #88 too.

@sfmig sfmig merged commit aa2fca4 into main Nov 22, 2023
6 checks passed
@samcunliffe
Copy link
Member

So for now I would suggest to merge this PR...

Agreed.

@sfmig sfmig deleted the smg/frame-extraction-w-reencoding branch July 15, 2024 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add SWC submission scripts to mainbranch of some version control, please 🥺
3 participants