[Backport 2024.2] fix(update_db_packages): fix issues with hanging threads during update #8082
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
update_db_packages procedure executes updating of scylla packages in threads for each node and uses sdcm.utils.decorators.retrying mechanism for retries. Through this decorator it is allowed for UnexpectedExit exception to occur up to the specified number of times, when installing packages, then the UnexpectedExit exception should be reraised. For some reason reraise doesn't return control back to the caller if update function is running in threading.Thread threads. In this case the thread where packages update is continously failing, and all retries are exhausted, hangs until test times out. The change switches from threading.Thread to concurrent.futures.ThreadPoolExecutor for parallelising packages update on nodes, which doesn't have issues with reraising exceptions in threads. (cherry picked from commit 076bf83)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
update_db_packages procedure executes updating of scylla packages in threads for each node and uses sdcm.utils.decorators.retrying mechanism for retries. Through this decorator it is allowed for UnexpectedExit exception to occur up to the specified number of times, when installing packages, then the UnexpectedExit exception should be reraised.
For some reason reraise doesn't return control back to the caller if update function is running in threading.Thread threads. So if for some reason a thread with packages update is continuously failing and all retries are exhausted, the thread hangs.
The change switches from threading.Thread to concurrent.futures.ThreadPoolExecutor for paralleling packages update on nodes. This approach doesn't have issues with reraising exceptions in threads.
Related tasks: #7162 and #7160
Testing
PR pre-checks (self review)
backportlabelsReminders
sdcm/sct_config.py)unit-test/folder)(cherry picked from commit 076bf83)
Parent PR: #7692