Shutdown exec on KubernetesTaskRunner stop #17601
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes a issue where the KubernetesTaskRunner still tracks task status after the overlord it is running on has lost leadership.
Description
The KubernetesTaskRunner doesn't actually do anything to stop watching running tasks in its stop method(), so the threads watching each task continue running doTask(). In this situation, the task queue is already shutdown, so the only thing that happens is the 'phantom' KubernetesTaskRunner sending notifications to its task queue that get ignored and potentially task logs getting double uploaded.
This is still not ideal, so we should shutdown the exec Executor in the stop method. This won't stop any of the tasks (they're running on k8s), but will stop the overlord's management of these tasks, which is the correct behavior.
Fixed the bug ...
Renamed the class ...
Added a forbidden-apis entry ...
Fixed a bug with overlord shutdown in mmless mode.
Release note
Bugfix with mmless ingestion.
Key changed/added classes in this PR
KubernetesTaskRunner
This PR has: