improving dynamic-sidecar design

[GitHK]

Click to expand old case hading

Listing of issues to address

dynamic sidecar service

• Data loss issues
  • dy-sidecar reboot volume/data issues osparc-simcore#3116
  • dy-sidecar & proxy restart policy osparc-simcore#3141
  • dy-sidecar started containers missing reboot options osparc-simcore#3142
• Make docker_compose_down reliable osparc-simcore#3172
• dy-sidecar issue: sometimes containers are created but not started osparc-simcore#3233
• dy-sidecar: sometimes containers are stuck in "created" state osparc-simcore#3247
• missing retry when uploading
  • when one upload through multipart uploads fails, all the upload fails osparc-simcore#3242
• use aiodocker and get a nice progress of the pulling osparc-simcore#3229
• dy-sidecar does not create all containers osparc-simcore#3293

director v2 service

• timeouts:
  • httpx client timing out after request finishes osparc-simcore#3091
  • timeout during service creation osparc-simcore#3111
  • Frontends marks services as failed #708
• resource management :
  • multiple zipping on same machine causes high I/O osparc-simcore#3143
• When dy-sidecar is in a failing state stop hammering requests osparc-simcore#3170
• sidecars cleanup:
  • director-v2 does not remove stuck sidecars after manual removal osparc-simcore#3241
  • failing or pending dy-sidecar are left and never removed even if deleted osparc-simcore#3219
  • Sidecars upload state when they failed and data is not there osparc-simcore#3266
• others:
  • ♻️ Dy-sidecar client unexpected error to be logged as an error osparc-simcore#3228

From users

• Service restart no closing the study
  • add "save" button #594
  • Option to relaunch a dynamic service without going to Dashboard #631
  • Add function to restart failed Service #630
• "Service Unavailable" in interactive nodes #600
• adding nodes only works sometimes #585

Uncategorised

• 🐛 : iSeg app/web are not in the same docker network osparc-simcore#3261
• After decompressing workspace in some cases files hang #659
• Prevent pulling anything inside dy-sidecar while a study is closing osparc-simcore#3240
• dynamic service save state fails with "can not write request body" #663
• dy-sidecar: save state fails with workspace.zip unavailable #664

Enhancements

• Closing study awaits for data to download before closing them osparc-simcore#3322

Continous testing

• E2E testing of large studies (open, close)

TODO

Preview Give feedback

1. Volume removal/agent lifecycle and autoscaling osparc-simcore#4084
   t:maintenance +1
   
2. agent removes orphaned stacks osparc-simcore#4498
   bug t:enhancement +2
   
3. Studies stay "closing" forever (=1h) when a dynamic service is pending waiting for available resources osparc-simcore#4442
   a:dynamic-sidecar bug +2
   
4. Saving state for services which did not start osparc-simcore#4441
   a:dynamic-sidecar bug +2
5. move image pulling from GC to osparc-agent osparc-simcore#4555
   +0
   
6. remove save_state when closing services osparc-simcore#4556
   +0

Quilmes

Preview Give feedback

1. S3TransferError 400 BadRequest with reason RequestTimeout osparc-simcore#3531
   a:dynamic-sidecar bug t:maintenance +3
   
2. Broken docker progress parsing when pulling images osparc-simcore#4440
   a:dynamic-sidecar bug +2
   

Baklava

Preview Give feedback

No tasks being tracked yet.

Sundae

Preview Give feedback

1. replace docker-compose 1.29.1 with docker cli and plugin osparc-simcore#4507
   a:dynamic-sidecar t:maintenance +2
   
2. When uploading chunks of a file (we gather all the tasks). If one fails then it is added to the bottom of the queue. Ideally it should be put at the beginning of the queue to be picked up as soon as possible. #974
   +0
   
3. dynamic-sidecar pulls inputs while user-services are starting osparc-simcore#4595
   bug +1
   

Watermelon

Preview Give feedback

1. When uploading chunks of a file (we gather all the tasks). If one fails then it is added to the bottom of the queue. Ideally it should be put at the beginning of the queue to be picked up as soon as possible. #974
   +0
   

Pastel de Nata

Preview Give feedback

1. volumes state is stored in the state volume inside the dy-sidecar #962
   +0
2. try to make the proxy start with the dy-sidecar by the director-v2 #963
   1 of 1
   +0
3. continue ✨ Add volume size limits for user services  osparc-simcore#3812 since there is a solution for it now

[pcrespov]

Two issues that IMO were missing above and were discussed during meeting (TODO: add reference to PR or oppened issues)

• we need to have the fix that cancels tasks in the dy-sidecar when the REST client timesout (currently all these task are accumulating) ✨ Is638/client cancellation osparc-simcore#3155
• we need to have the fix in the dy-sidecar that prevents running XX docker-compose CLI calls in parallel (currently all these run in parallel and collapses the host RAM/CPUs) 🐛♻️ Is638/fixes docker-compose operations bursts and create containers call in dy-sidecar osparc-simcore#3187

[GitHK]

Update on sprint [Diolkos]

Done

• Helps when opening sutdy:
  ♻️ refactor dy-sidecar volume searching osparc-simcore#3110
  🐛 adds container cleanup before starting and after closing osparc-simcore#3153

• Helps when closing and opening study:
  ♻️ refactor dv-2 dy-sidecar's API client osparc-simcore#3121

• Others:
  ♻️🐛 Constraint dynamic-sidecar to spawned node after creation osparc-simcore#3104
  ♻️🔨 Is638/dynamic sidecar refactoring (round 1) osparc-simcore#3147
  ♻️ ✨ Is638/adds CLI to dynamic-sidecar (round 2) osparc-simcore#3152
  ♻️ Is638/dynamic sidecar refactors tests (round 3) osparc-simcore#3154
  🚑️ Is638/tmp disable cancellation  osparc-simcore#3161

Ongoing

• Helps when closing study:
  ♻️ limit rclone CPU and memory usage osparc-simcore#3080
  (⚠️ devops) ♻️✨Adding distributed locking to throttle concurrent saves on nodes osparc-simcore#3160

• Helps when opening study:
  WIP: 🐛 is638/docker-compose bursts osparc-simcore#3157

Open

• dy-sidecar reboot volume/data issues osparc-simcore#3116 between reboots dy-sidecar looses original volumes
• multiple zipping on same machine causes high I/O osparc-simcore#3143
• httpx client timing out after request finishes osparc-simcore#3091
• timeout during service creation osparc-simcore#3111
• ✨ Is638/client cancellation osparc-simcore#3155
• dy-sidecar started containers missing reboot options osparc-simcore#3142
• dy-sidecar & proxy restart policy osparc-simcore#3141

User requests

• "Service Unavailable" in interactive nodes #600
• adding nodes only works sometimes #585
• add "save" button #594
• Option to relaunch a dynamic service without going to Dashboard #631
• Add function to restart failed Service #630

[GitHK]

Update on sprint Pastel de Nata

Done:

• 🐛 No longer upload output ports when study is in read only mode osparc-simcore#4207
• ✨ Add volume size limits for user services  osparc-simcore#3812
• ✨Saving volume stats inside the shared store volume osparc-simcore#4267
• ✨Propagate key and version to dy-sidecar service labels osparc-simcore#4249
• ✨Propagate capabilities to correct processes osparc-simcore#4225

Ongoing:

• under review ♻️ Start the dy-proxy with the dy-sidecar osparc-simcore#4291

[GitHK]

Update Watermelon

no progress

[GitHK]

Update Sundae

Done:

• ♻️🐛 dynamic-sidecar will ignore inputs pull commands while containers are created osparc-simcore#4601
• 🐛 fix dynamic-sidecar outputs watcher osparc-simcore#4600
• 🐛 fix issue with parsing shared store osparc-simcore#4485
• ♻️ extending run_id with int_timestamp osparc-simcore#4505
• ⬆️ upgrading docker-compose to 2.x.x in dynamic-sidecar 🚨 osparc-simcore#4545
• ⬆️ upgrade dynamic-sidecar service requirements osparc-simcore#4551