Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prep for SkyDriver v2 (EWMS v1) [minor] #273

Draft
wants to merge 243 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
243 commits
Select commit Hold shift + click to select a range
3ba7708
remove more client starter remnants
ric-evans Aug 21, 2024
7b1e895
<bot> update setup.cfg
wipacdevbot Aug 21, 2024
7e69286
<bot> update dependencies*.log files(s)
wipacdevbot Aug 21, 2024
b99d478
make `SKYSCAN_SKYDRIVER_SCAN_ID` required
ric-evans Aug 21, 2024
a36c70c
make `SKYSCAN_SKYDRIVER_SCAN_ID` required - tests
ric-evans Aug 21, 2024
ee95c9f
make `SKYSCAN_SKYDRIVER_SCAN_ID` required - tests - 2
ric-evans Aug 21, 2024
84c42f9
make `SKYSCAN_SKYDRIVER_SCAN_ID` required - tests - 3
ric-evans Aug 21, 2024
f0d86d1
env vars: backward-compatible defaults/overrides
ric-evans Aug 21, 2024
4ee5f6e
env vars: backward-compatible defaults/overrides - 2
ric-evans Aug 21, 2024
c56f3d0
fix client
ric-evans Aug 22, 2024
c94b57f
add `ewms-scan.sh` - wip
ric-evans Aug 22, 2024
ae5431a
parse startup.json in reco_icetray.py (prev client.py)
ric-evans Aug 23, 2024
6e2bef5
add `ewms-scan.sh` - wip 2
ric-evans Aug 23, 2024
116fb73
parse startup.json in reco_icetray.py (prev client.py) - 2
ric-evans Aug 23, 2024
a9c4fb9
rm extra file
ric-evans Aug 23, 2024
c3218af
update `test-run-single-pixel`
ric-evans Aug 23, 2024
6b910fe
remove `test-build-docker` as dep
ric-evans Aug 23, 2024
52979ad
remove `test-build-docker` as dep - 2
ric-evans Aug 23, 2024
8a66dd2
fix call to `i3live_json_to_frame_packet`
ric-evans Aug 23, 2024
d38c809
typo
ric-evans Aug 23, 2024
a644d3a
logging
ric-evans Aug 23, 2024
6807895
logging - 2
ric-evans Aug 23, 2024
7c169d2
init attr + s3 env vars
ric-evans Sep 11, 2024
99fa1de
chmod
ric-evans Sep 11, 2024
4453d0e
update default platform in cvmfs build
ric-evans Sep 11, 2024
71fc847
update default platform in cvmfs build - 2
ric-evans Sep 11, 2024
bf74f83
<bot> update dependencies*.log files(s)
wipacdevbot Sep 11, 2024
fbe1982
misc updates
ric-evans Sep 12, 2024
b5de511
<bot> update dependencies*.log files(s)
wipacdevbot Sep 12, 2024
544cc25
cl args: mkdir if not exits (prev error)
ric-evans Sep 12, 2024
327ab69
<bot> update dependencies*.log files(s)
wipacdevbot Sep 12, 2024
771d8c2
misc script updates
ric-evans Sep 12, 2024
7dd4de3
fix init container args
ric-evans Sep 16, 2024
02b2d88
unpin oms deps
ric-evans Sep 16, 2024
6cab0c3
<bot> update setup.cfg
wipacdevbot Sep 16, 2024
232cbec
<bot> update dependencies*.log files(s)
wipacdevbot Sep 16, 2024
4523a06
must pickle now
ric-evans Sep 17, 2024
cd2a4d2
pickle within a json (test files will need to change)
ric-evans Sep 17, 2024
f843921
remove overcomplicated "pull policy" var
ric-evans Sep 18, 2024
fcb8428
<bot> update README.md
wipacdevbot Sep 18, 2024
f0702f3
<bot> update dependencies*.log files(s)
wipacdevbot Sep 18, 2024
d5dde26
require mq env vars
ric-evans Sep 18, 2024
85e0e6c
<bot> update README.md
wipacdevbot Sep 18, 2024
60996e7
require mq env vars - 2
ric-evans Sep 18, 2024
618c8a8
typo fix
ric-evans Sep 18, 2024
2748a20
wip run pilot
ric-evans Sep 18, 2024
20bd051
another go at serialization
ric-evans Sep 18, 2024
1d297d8
another go at serialization - 2
ric-evans Sep 18, 2024
3d35270
try speeding up free-disk-space
ric-evans Sep 18, 2024
e1b63c3
fix env var forwarding
ric-evans Sep 18, 2024
1c24b98
remove ewms_pilot dep & client.py
ric-evans Sep 18, 2024
2c824fb
<bot> update setup.cfg
wipacdevbot Sep 18, 2024
fe3ab3f
remove ewms_pilot dep & client.py - 2
ric-evans Sep 18, 2024
404429d
<bot> update dependencies*.log files(s)
wipacdevbot Sep 18, 2024
da85941
env vars don't need vals for docker `--env`
ric-evans Sep 18, 2024
43c4eee
set N_CLIENTS to 1 for now
ric-evans Sep 18, 2024
5707131
`EWMS_PILOT_*` vars
ric-evans Sep 18, 2024
ee0a2cf
misc tee
ric-evans Sep 19, 2024
936042c
use pyenv for pilot instead of docker
ric-evans Sep 19, 2024
c287392
refactor serialization helpers
ric-evans Sep 19, 2024
b726253
`pip install virtualenv`
ric-evans Sep 19, 2024
7c92006
squash merge #271
ric-evans Sep 24, 2024
b5281e7
<bot> update dependencies*.log files(s)
wipacdevbot Sep 24, 2024
c855f1f
fixing single pix test
ric-evans Sep 24, 2024
8a43e98
fix import
ric-evans Sep 24, 2024
e9f7e46
another merge fix
ric-evans Sep 24, 2024
02f2184
`python -m ewms_pilot[rabbitmq]`
ric-evans Sep 24, 2024
346fcf4
`pip install ewms_pilot[rabbitmq]`
ric-evans Sep 24, 2024
ab2d401
test: use env var for docker tag
ric-evans Sep 24, 2024
8d1e4a1
test: use env var for docker tag - 2
ric-evans Sep 24, 2024
7b67db7
set pilot root dir
ric-evans Sep 24, 2024
92e0bf0
use pilot's default for stopping when task errors
ric-evans Sep 25, 2024
78e7cd1
put back the docker step as dep, bump other actions
ric-evans Sep 25, 2024
effe3c2
put back the docker step as dep - 2
ric-evans Sep 25, 2024
fe9d3ab
put back the docker step as dep - 3
ric-evans Sep 25, 2024
38c26fe
use artifacts for docker persistence - 1
ric-evans Sep 25, 2024
74a5d9c
use artifacts for docker persistence - 2
ric-evans Sep 25, 2024
1ef65ae
use artifacts for docker persistence - 3
ric-evans Sep 25, 2024
b15d514
use artifacts for docker persistence - 4
ric-evans Sep 25, 2024
6d1e2dc
use artifacts for docker persistence - 5 (remove)
ric-evans Sep 25, 2024
fe171a5
fix file types
ric-evans Sep 25, 2024
4f33d55
use artifacts for docker persistence - 6 (remove)
ric-evans Sep 25, 2024
2956b73
mkdir
ric-evans Sep 25, 2024
f8e5fef
decrease server timeout 10x
ric-evans Sep 25, 2024
81bb8be
don't make .dockerbuild artifact
ric-evans Sep 25, 2024
5e5fe44
fix ci flags
ric-evans Sep 25, 2024
5215a91
fix startup.json handling
ric-evans Sep 25, 2024
dab753d
fix startup.json handling - 2
ric-evans Sep 25, 2024
aad4157
fix startup.json handling - 3
ric-evans Sep 26, 2024
ce58195
fix startup.json handling - 4
ric-evans Sep 26, 2024
bc5672b
no artifact compression
ric-evans Sep 26, 2024
11ed5c1
(debug)
ric-evans Sep 26, 2024
9f8b6b2
Revert "no artifact compression"
ric-evans Sep 26, 2024
26186de
sorting out dirpaths
ric-evans Sep 26, 2024
b2de4bf
sorting out dirpaths - 2
ric-evans Sep 26, 2024
3a9639a
sorting out dirpaths - 3
ric-evans Sep 26, 2024
f4ef8fb
speed up free-disk
ric-evans Sep 26, 2024
55dc511
dumping logs
ric-evans Sep 26, 2024
9abdcf5
dumping logs - 2
ric-evans Sep 26, 2024
6f87ff0
dumping logs - 3
ric-evans Sep 26, 2024
899f8f9
dumping logs - 4
ric-evans Sep 26, 2024
36de88c
dumping logs - 5
ric-evans Sep 26, 2024
b0182d4
syntax fix
ric-evans Sep 26, 2024
ea803be
dumping logs - 6
ric-evans Sep 26, 2024
cc306af
gha logic
ric-evans Sep 26, 2024
67997f5
free less disk
ric-evans Sep 26, 2024
f04a8d0
singularity test: `_EWMS_PILOT_CONTAINER_PLATFORM: apptainer`
ric-evans Sep 27, 2024
a819bf0
split up stderr / stdout dumping
ric-evans Sep 27, 2024
b538ea5
refactoring test dirpaths
ric-evans Sep 27, 2024
06ec8e0
call them "workers" and the *task* a "client"
ric-evans Sep 27, 2024
4a526c0
fix log dumping
ric-evans Sep 27, 2024
72b2f21
refactoring test dirpaths - 2
ric-evans Sep 27, 2024
ef62c19
rename step
ric-evans Sep 27, 2024
8e00bd9
rename step - 2
ric-evans Sep 27, 2024
1901674
rename step - 3
ric-evans Sep 27, 2024
ee2385d
improve wait-for-scan logic
ric-evans Sep 27, 2024
1f3cece
improve wait-for-scan logic - 2
ric-evans Sep 27, 2024
80f04fe
improve wait-for-scan logic - 3
ric-evans Sep 27, 2024
a860014
crash dummy: don't stop on first task fail
ric-evans Sep 27, 2024
07a3ba0
improve wait-for-scan logic - 4
ric-evans Sep 27, 2024
5e1a499
timeouts
ric-evans Sep 27, 2024
c764dc9
fix crash dummy check
ric-evans Sep 27, 2024
775bc09
remove `tee` from client parts
ric-evans Sep 27, 2024
a0a167b
remove `tee` from client parts - 1
ric-evans Sep 27, 2024
0cb2c38
fix crash dummy check - 2
ric-evans Sep 27, 2024
3f0dbf8
get hacky with `--shm-size=6gb`
ric-evans Sep 27, 2024
9419220
(debug)
ric-evans Sep 27, 2024
277c57a
(debug - 2)
ric-evans Sep 27, 2024
0c924c1
take 2 at wait logic
ric-evans Sep 27, 2024
cfa87ee
bump EWMS_PILOT_TASK_TIMEOUT to 60
ric-evans Sep 27, 2024
cfa643d
take 2 at wait logic - 2
ric-evans Sep 27, 2024
80eebbc
take 2 at wait logic - 3
ric-evans Sep 27, 2024
dc604bc
crank up EWMS_PILOT_TASK_TIMEOUT
ric-evans Sep 27, 2024
2692f68
Merge branch 'main' into prep-skydriver-2.0-2
ric-evans Sep 30, 2024
a3a8c45
<bot> update dependencies*.log files(s)
wipacdevbot Sep 30, 2024
370b9c8
(temp: remove ci timeouts)
ric-evans Oct 1, 2024
1564d3e
<bot> update dependencies*.log files(s)
wipacdevbot Oct 1, 2024
d0c7144
ci: inc `SKYSCAN_MQ_TIMEOUT_FROM_CLIENTS`
ric-evans Oct 1, 2024
9b36003
Merge remote-tracking branch 'origin/prep-skydriver-2.0-2' into prep-…
ric-evans Oct 1, 2024
5d4a44e
Merge branch 'main' into prep-skydriver-2.0-2
ric-evans Oct 2, 2024
a21c2d8
ok, so not much time was actually saved using artifacts
ric-evans Oct 2, 2024
b8dcd5f
inc `SKYSCAN_MQ_TIMEOUT_FROM_CLIENTS`
ric-evans Oct 2, 2024
0fde6f5
fix docker load
ric-evans Oct 2, 2024
fd43f99
inc `EWMS_PILOT_TASK_TIMEOUT`
ric-evans Oct 2, 2024
44e393e
ci: set `_EWMS_PILOT_DOCKER_SHM_SIZE`
ric-evans Oct 2, 2024
bc960b7
comment
ric-evans Oct 2, 2024
293bf79
(test: use `{{DATA_HUB}}`)
ric-evans Oct 3, 2024
8a2037f
use `EWMS_PILOT_EXTERNAL_DIRECTORIES` for startup.json
ric-evans Oct 3, 2024
a490c64
remove unused code
ric-evans Oct 3, 2024
0e3473c
rm `launch_server.sh`, move to `local-scan.sh`
ric-evans Oct 3, 2024
ecae72b
finally fix singularity test
ric-evans Oct 3, 2024
f962ec6
revert accidental dockerfile change
ric-evans Oct 3, 2024
54d55be
bump to 2 workers in ci
ric-evans Oct 3, 2024
ac956fc
bump to 2 workers in ci - 2 (singularity)
ric-evans Oct 3, 2024
25773c1
bump to 4 workers in ci
ric-evans Oct 4, 2024
4ae6a4b
Revert "bump to 4 workers in ci"
ric-evans Oct 4, 2024
20a0222
<bot> update dependencies*.log files(s)
wipacdevbot Oct 4, 2024
5a8a4d7
Merge branch 'main' into prep-skydriver-2.0-2
ric-evans Oct 4, 2024
06c25fa
massively trim down README (to be added to later)
ric-evans Oct 4, 2024
be6dc29
shake up and settle the ci env vars
ric-evans Oct 4, 2024
8d39eb0
<bot> update README.md
wipacdevbot Oct 4, 2024
bed102b
set task timeout to 10 mins
ric-evans Oct 4, 2024
02892ad
Revert "(temp: remove ci timeouts)"
ric-evans Oct 4, 2024
a78877e
adjust ci timeouts
ric-evans Oct 4, 2024
df26917
comment
ric-evans Oct 4, 2024
82a3496
<bot> update dependencies*.log files(s)
wipacdevbot Oct 9, 2024
4f85b0a
comment
ric-evans Oct 14, 2024
1fb99e2
(test task timeouts)
ric-evans Oct 14, 2024
d157b56
<bot> update dependencies*.log files(s)
wipacdevbot Oct 14, 2024
25d5c1b
set
ric-evans Oct 14, 2024
23e4a5b
set `EWMS_PILOT_OKAY_ERRORS`
ric-evans Oct 14, 2024
993810e
Merge remote-tracking branch 'origin/prep-skydriver-2.0-2' into prep-…
ric-evans Oct 14, 2024
ffe217d
set `EWMS_PILOT_OKAY_ERRORS` - 2
ric-evans Oct 14, 2024
cb8b54f
better wait logic
ric-evans Oct 14, 2024
1b283bd
better wait logic - 2 (sleeps)
ric-evans Oct 14, 2024
170e011
echo
ric-evans Oct 14, 2024
e00ee53
(debug)
ric-evans Oct 14, 2024
1265b6a
set 'EWMS_PILOT_TASK_TIMEOUT' for reco algo + event file
ric-evans Oct 15, 2024
7716560
syntax
ric-evans Oct 15, 2024
2c1f85f
echo
ric-evans Oct 15, 2024
451c832
abandon task timeout idea
ric-evans Oct 15, 2024
11b46ad
light refactor reporter.py
ric-evans Oct 15, 2024
d89a67a
use a moving window to predict total runtimes
ric-evans Oct 16, 2024
f2e52b9
mypy
ric-evans Oct 16, 2024
ced53c8
fix units
ric-evans Oct 16, 2024
7421490
fix units - 2
ric-evans Oct 16, 2024
fe0a2b4
comment
ric-evans Oct 16, 2024
d6de8b5
fix math
ric-evans Oct 16, 2024
0f96957
use `SKYSCAN_PROGRESS_RUNTIME_PREDICTION_WINDOW_MIN`
ric-evans Oct 16, 2024
4604e0f
`ewms-scan.sh`: pip installs
ric-evans Oct 16, 2024
6183524
`ewms-scan.sh`: tee scanner output
ric-evans Oct 16, 2024
70eb936
`ewms-scan.sh`: misc
ric-evans Oct 16, 2024
403c0c1
Merge branch 'main' into prep-skydriver-2.0-2
ric-evans Oct 17, 2024
b789448
exit cleanup
ric-evans Oct 24, 2024
cda2c18
<bot> update dependencies*.log files(s)
wipacdevbot Oct 24, 2024
1c54171
dockerfile syntax
ric-evans Oct 24, 2024
c06194f
misc script updates
ric-evans Oct 25, 2024
8f8aa9a
env vars
ric-evans Oct 25, 2024
44b1432
use `{{DATA_HUB}}`
ric-evans Oct 28, 2024
2de5090
script: set file extensions
ric-evans Oct 28, 2024
d800f85
misc linter ignores
ric-evans Oct 28, 2024
3105f48
misc linter ignores - 2
ric-evans Oct 28, 2024
fb31b71
update .gitignore
ric-evans Oct 28, 2024
b844dab
remove ENV from `recos/`
ric-evans Oct 28, 2024
ac245d3
remove ENV from `recos/` - 2
ric-evans Oct 28, 2024
900ae0b
move `ENV` to `server/`
ric-evans Oct 28, 2024
695019e
<bot> update dependencies*.log files(s)
wipacdevbot Oct 28, 2024
2407c9d
decrease diffs
ric-evans Oct 28, 2024
b25daf9
fix loggers
ric-evans Oct 28, 2024
763c364
fix loggers - 2
ric-evans Oct 28, 2024
72a9619
fix loggers - 3
ric-evans Oct 28, 2024
c5a2d7e
give `client/` its own `ENV`
ric-evans Oct 28, 2024
7d82863
fix client `main.py`
ric-evans Oct 28, 2024
218acf7
script: indenting
ric-evans Oct 29, 2024
04e3b50
proper quoting (tms uses shlex now)
ric-evans Oct 29, 2024
c230d95
script: error handling + send finish call
ric-evans Oct 30, 2024
d4b2029
echo formatting
ric-evans Oct 30, 2024
d5821bb
echo formatting - 2
ric-evans Oct 30, 2024
ed76ab9
add/combine tests into `test-run-dummy-recos`
ric-evans Oct 31, 2024
60f2747
fix ci - set/forward env vars
ric-evans Oct 31, 2024
2f53105
misc
ric-evans Nov 7, 2024
ef147e0
fix bash traps; adjust timeouts, memory
ric-evans Nov 7, 2024
1042179
org the configs
ric-evans Nov 7, 2024
ee3fc1c
script: timeouts
ric-evans Nov 7, 2024
b9eb166
misc
ric-evans Nov 7, 2024
6b3174d
misc - 2
ric-evans Nov 7, 2024
a2942c5
misc - 3
ric-evans Nov 7, 2024
57187f8
trap fix
ric-evans Nov 7, 2024
01c8bcc
add "technique" to "predictions" obj
ric-evans Nov 7, 2024
103adaf
bump window to `0.25`
ric-evans Nov 7, 2024
303f775
bump window min to 500
ric-evans Nov 7, 2024
1edc096
add `"technique"` to `"predictions"` in reporter
ric-evans Nov 7, 2024
59df77b
script: add PREDICTIVE_SCANNING_THRESHOLD
ric-evans Nov 7, 2024
7af73f4
script: add PREDICTIVE_SCANNING_THRESHOLD - 2
ric-evans Nov 7, 2024
9f7aa96
ci: fix apptainer tests
ric-evans Nov 7, 2024
4895f7b
fix prediction technique string
ric-evans Nov 7, 2024
013d2a3
update prediction technique string - 2
ric-evans Nov 7, 2024
ad91912
update prediction technique string - 3
ric-evans Nov 7, 2024
7aaae64
adjust report predictions settings
ric-evans Nov 7, 2024
0bf0c67
adjust report predictions settings - 2
ric-evans Nov 7, 2024
8730c58
<bot> update dependencies*.log files(s)
wipacdevbot Nov 11, 2024
2b282be
test `WIPACrepo/wipac-dev-py-versions-action@from-args-only`
ric-evans Nov 14, 2024
1f6535c
use `WIPACrepo/[email protected]`
ric-evans Nov 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
inputs:
platform:
description: 'Platform'
default: 'linux/amd64,linux/arm64/v8'
default: 'linux/amd64'
required: false
delete:

Expand Down Expand Up @@ -67,7 +67,7 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

-
-
name: Push Docker Image X86_64
# if: ${{ arch != 'arm' }}
uses: docker/build-push-action@v3
Expand All @@ -78,7 +78,7 @@ jobs:
tags: ${{ steps.docker_meta.outputs.tags }}
labels: ${{ steps.docker_meta.outputs.labels }}

# -
# -
# name: Push Docker Image
# if: ${{ arch == 'arm' }}
# uses: docker/build-push-action@v3
Expand Down Expand Up @@ -132,4 +132,4 @@ jobs:
uses: WIPACrepo/[email protected]
with:
github_token: ${{ secrets.PERSONAL_ACCESS_TOKEN }} # so job can git push
remove_regex_path: 'realtime/${{ github.event.ref }}.*'
remove_regex_path: 'realtime/${{ github.event.ref }}.*'
Loading
Loading