Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make deploy.RunPod wait for pod readiness rather than running phase #346

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

nadiamoe
Copy link
Member

@nadiamoe nadiamoe commented Sep 14, 2023

Description

This PR changes the WaitPodRunning function to WaitPodReady, which checks whether the pod has the Ready condition set to True instead of simply being in the Running phase.

If a pod does not contain any container with a readiness probe defined, WaitPodReady should behave exactly the same as WaitPodRunning did. However, if one or more containers have a readiness probe, WaitPodReady will wait until they are satisfied.

This is helpful to reduce flakiness on certain tests which previously attempted to initiate a connection to a pod once it was running, but before the pod had time to actually get started. By attaching a readiness probe to those pods, this scenario can be avoided, as deploy.RunPod will wait or error until the readiness probe passes.

This PR also adds a helper WithHTTPReadinessProbe function to ContainerBuilder, that attaches an HTTP GET readiness probe to a container with reasonable defaults.

Checklist:

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works.
  • I have run linter locally (make lint) and all checks pass.
  • I have run tests locally (make test) and all tests pass.
  • I have run relevant integration test locally (make integration-xxx for affected packages)
  • I have run relevant e2e test locally (make e2e-xxx for disruptors, or cluster related changes)
  • Any dependent changes have been merged and published in downstream modules

@nadiamoe nadiamoe changed the title K8s wait ready Make deploy.RunPod wait for pod readiness rather than running phase Sep 14, 2023
@nadiamoe nadiamoe force-pushed the k8s-wait-ready branch 2 times, most recently from 4e7747e to 4f19c30 Compare September 14, 2023 15:47
@pablochacin pablochacin marked this pull request as ready for review November 13, 2024 16:12
@pablochacin pablochacin requested a review from a team as a code owner November 13, 2024 16:12
@pablochacin pablochacin requested review from szkiba and pablochacin and removed request for a team and szkiba November 13, 2024 16:12
@pablochacin pablochacin marked this pull request as draft November 13, 2024 16:12
@pablochacin pablochacin marked this pull request as ready for review November 13, 2024 17:03
@pablochacin pablochacin self-requested a review November 14, 2024 14:05
Copy link
Collaborator

@pablochacin pablochacin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the need for the BuildNginxPodWithoutProbes() fixture for the disruptor tests a design smell. The idea of introducing a readiness probe in the test fixtures was to avoid the flakiness of certain tests, but it introduces more subtle problems as the disruptor tests will fail if this non-intuitive fixture is used.

The reason for these potential test failures is that the tests apply the disruption to all requests to a target pod or service. This includes the readiness/liveness probes if any, unless explicitly excluded. But which probes to exclude may not be obvious and depended on the fixture's definition

I think that either we reverse the situation and make the alternative BuildxxxWithProbes fixture or we address the problem of automatically excluding the pod's probes from disruptions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants