some pods that are stuck in "pending" state after deployment with Kubernetes #423
Assignees
Labels
Comments
|
@antonadamov , is your problem still actual? |
|
[2024-09-16 15:06:01] Waiting for job mgp-tip-report-portal-reportportal-migrations... these logs are from the "migrations-waiting-init" pod in one of the pending containers. all pending containers have this pod repeating the same message wait-for-it.sh: waiting for mgp-tip-report-portal-postgresql.default.svc.cluster.local:5432 without a timeout these are the only logs from the migrations container |
|
@antonadamov, please provide more information: are you trying to install with external Postgres? Have you installed Ingress Controller https://reportportal.io/docs/installation-steps-advanced/ComponentsOverview#gateway ? |
|
@Pink-Bumblebee , @c0demonky worked on the deployment. |
|
@Pink-Bumblebee we are running this service on an enterprise eks node. I created a dns record and cname to route requests to the services, which are reachable but showing 503 -- aside from UI and Index. |
|
Hi! Regarding our workflow, the Helm Chart first deploys PostgreSQL, RabbitMQ, and OpenSearch. Next, the Migration service prepares the database and OpenSearch. Once the migration is complete, the remaining services will start running. Could you please share the status and logs of all services? This will help me understand in which step we have a problem. |
|
Migrations Analyzer, Analyzer Train, Metrics Gatherer, OpenSearch, Jobs, UAT, API Defaulted container "analyzer" out of: analyzer, migrations-waiting-init (init) Postgres RabbitMQ MinIO |
|
@Pink-Bumblebee im unable to upload files, so the index and ui logs cant be posted here. they're fairly long text files. but, we are able to access the ui and attempt to login. 
|
|
@hlebkanonik , any idea? |
|
Can you please confirm the status of the migrations pod? Has it Succeeded? |
|
What you see in the logs is all the information we have. here are those logs again Pod is running |
|
Can you connect to the database and check if something exists there? For example, Have you used our latest release, version 24.1.7? |
|
how can i access the database? through shell only? yes, using 24.1.7. i just redeployed it today. Commands: |
|
@c0demonky , you could use external tools like DBeaver. Standard port is 5432. Username and password - from values.yaml or ask your admin. |
|
Shell: kubectl exec -it <DB POD NAME> -n <NAMESPACE> -- bash
> psql -U postgres -h localhost -d reportportal
> SELECT * FROM public.schema_migrations; |
|
Output executing the above commands: |
|
Which version of Kubernetes are you using? It seems like Helm webhooks are not working properly. You can manually delete the migrations pod, all the services should run well. |
|
Hi! Any updates? |
|
I was also running into this issue deploying 24.2 using the helm charts. I tried deleting the pod, and it just got re-created and the logs report "no change". I am using OpenShift 4.14 and the output of My quick hack to fix it was to delete the migration job itself, then had to go through all the deployments and delete the init containers and restart the deployments. Note: |
|
Hi @calebevans, this might be an OpenShift issue because we tested it on a stock OpenShift cluster, and it worked fine except for the Routes. We’ll roll out a Routes update in the next release. Have you used the service account we provided to work with the ReportPortal? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
could you please help with an issue I'm facing after deploying RP using the Helm chart? After running Installing the Chart
commands, multiple pods are stuck in "pending" state. There are no resource issues on the EKS cluster. pods that are stuck:
uat, analyzer, api, jobs, metrics, and opensearch cluster master
The text was updated successfully, but these errors were encountered: