frequent sentry worker restart 26.7.0

[theechofive]

Release: https://github.com/sentry-kubernetes/charts/releases/tag/sentry-v26.7.0
Sentry installed via ArgoCD
The chart is in my repository and ArgoCD deploys it to the cluster.
Sentry ArgoCD application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sentry
spec:
  destination:
    name: ''
    namespace: sentry
    server: 'https://kubernetes.default.svc'
  source:
    path: sentry
    repoURL: 'git@*GIT_URL*/cluster.git'
    targetRevision: master
    helm:
      valueFiles:
        - values.yaml
  sources: []
  project: default
  syncPolicy:
    syncOptions:
      - PruneLast=true
      - RespectIgnoreDifferences=true
      - ServerSideApply=true
      - ApplyOutOfSyncOnly=true
      - CreateNamespace=true
ignoreDifferences:
  - kind: Secret
    name: sentry-kafka-kraft-cluster-id
    namespace: sentry
    jsonPointers:
      - /data
  - kind: Secret
    name: sentry-nginx-tls
    namespace: sentry
    jsonPointers:
      - /data
  - kind: Secret
    name: sentry-sentry-postgresql
    namespace: sentry
    jsonPointers:
      - /data         
  - group: apps
    kind: StatefulSet
    name: sentry-clickhouse
    namespace: sentry
    jsonPointers:
      - /spec/volumeClaimTemplates
  - group: apps
    kind: StatefulSet
    name: sentry-sentry-postgresql
    namespace: sentry
    jsonPointers:
      - /spec/volumeClaimTemplates
  - group: apps
    kind: StatefulSet
    name: sentry-sentry-redis-master
    namespace: sentry
    jsonPointers:
      - /spec/volumeClaimTemplates
  - group: apps
    kind: StatefulSet
    name: sentry-sentry-redis-replicas
    namespace: sentry
    jsonPointers:
      - /spec/volumeClaimTemplates
  - group: apps
    kind: StatefulSet
    name: sentry-zookeeper-clickhouse
    namespace: sentry
    jsonPointers:
      - /spec/volumeClaimTemplates

Describe

Name:             sentry-worker-7ffbc46c4-d4x7c
Namespace:        sentry
Priority:         0
Service Account:  default
Node:             pool-xjrh31v09-gpinh/10.110.0.3
Start Time:       Sun, 01 Dec 2024 13:42:22 +0200
Labels:           app=sentry
                  pod-template-hash=7ffbc46c4
                  release=sentry
                  role=worker
Annotations:      checksum/config.yaml: 4248f00b8cdb864d72ed09827a257af9a8aea080150de19be494a98dfad19adf
                  checksum/configYml: 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
                  checksum/sentryConfPy: 989bd5b88049cbfb5c9de170a98754359c4580b9ff316a9727d3fc7cb2383c3c
Status:           Running
IP:               10.244.6.52
IPs:
  IP:           10.244.6.52
Controlled By:  ReplicaSet/sentry-worker-7ffbc46c4
Containers:
  sentry-worker:
    Container ID:  containerd://d1265981838e0c485c896d9cc49491ce6744a9832f6cca374a373b9699bd940c
    Image:         getsentry/sentry:24.9.0
    Image ID:      docker.io/getsentry/sentry@sha256:1830c64c38383ff8e317bc7ba2274d27d176a113987fdc67e8a5202a67a70bad
    Port:          <none>
    Host Port:     <none>
    Command:
      sentry
    Args:
      run
      worker
    State:          Running
      Started:      Mon, 02 Dec 2024 17:19:37 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 02 Dec 2024 17:11:36 +0200
      Finished:     Mon, 02 Dec 2024 17:14:36 +0200
    Ready:          True
    Restart Count:  304
    Liveness:       exec [sentry exec -c from sentry.celery import app; import os; dest="celery@{}".format(os.environ["HOSTNAME"]); print(app.control.ping(destination=[dest], timeout=5)[0][dest]["ok"])] delay=10s timeout=10s period=60s #success=1 #failure=3
    Environment:
      C_FORCE_ROOT:       true
      SNUBA:              http://sentry-snuba:1218
      VROOM:              http://sentry-vroom:8085
      SENTRY_SECRET_KEY:  <set to the key 'key' in secret 'sentry-sentry-secret'>                    Optional: false
      POSTGRES_PASSWORD:  <set to the key 'postgres-password' in secret 'sentry-sentry-postgresql'>  Optional: false
      POSTGRES_USER:      postgres
      POSTGRES_NAME:      sentry
      POSTGRES_HOST:      sentry-sentry-postgresql
      POSTGRES_PORT:      5432
    Mounts:
      /etc/sentry from config (ro)
      /var/lib/sentry/files from sentry-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pv6h2 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sentry-sentry
    Optional:  false
  sentry-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-pv6h2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  7m19s (x910 over 27h)   kubelet  Liveness probe failed: command timed out: "sentry exec -c from sentry.celery import app; import os; dest=\"celery@{}\".format(os.environ[\"HOSTNAME\"]); print(app.control.ping(destination=[dest], timeout=5)[0][dest][\"ok\"])" timed out after 10s
  Warning  BackOff    2m19s (x3641 over 27h)  kubelet  Back-off restarting failed container sentry-worker in pod sentry-worker-7ffbc46c4-d4x7c_sentry(6e112fc5-b061-4522-8b43-3bb5ce553cf9)

Logs

15:01:01 [INFO] sentry.bgtasks: bgtask.spawn (task_name='sentry.bgtasks.clean_dsymcache:clean_dsymcache')
15:01:01 [INFO] sentry.bgtasks: bgtask.spawn (task_name='sentry.bgtasks.clean_releasefilecache:clean_releasefilecache')
/.venv/lib/python3.12/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
  warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
 -------------- celery@sentry-worker-7ffbc46c4-d4x7c v5.3.5 (emerald-rush)
--- ***** -----
-- ******* ---- Linux-6.1.0-25-amd64-x86_64-with-glibc2.36 2024-12-02 15:01:09
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         sentry:0x7f6d62239e80
- ** ---------- .> transport:   amqp://guest:**@sentry-rabbitmq:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> activity.notify  exchange=default(direct) key=activity.notify
                .> alerts           exchange=default(direct) key=alerts
                .> app_platform     exchange=default(direct) key=app_platform
                .> app_platform.control exchange=default(direct) key=app_platform.control
                .> appstoreconnect  exchange=default(direct) key=sentry.tasks.app_store_connect.#
                .> assemble         exchange=default(direct) key=assemble
                .> auth             exchange=default(direct) key=auth
                .> auth.control     exchange=default(direct) key=auth.control
                .> auto_enable_codecov exchange=default(direct) key=auto_enable_codecov
                .> auto_resolve_issues exchange=default(direct) key=auto_resolve_issues
                .> auto_transition_issue_states exchange=default(direct) key=auto_transition_issue_states
                .> backfill_seer_grouping_records exchange=default(direct) key=backfill_seer_grouping_records
                .> buffers.incr     exchange=default(direct) key=buffers.incr
                .> buffers.process_pending exchange=default(direct) key=buffers.process_pending
                .> buffers.process_pending_batch exchange=default(direct) key=buffers.process_pending_batch
                .> check_new_issue_threshold_met exchange=default(direct) key=check_new_issue_threshold_met
                .> cleanup          exchange=default(direct) key=cleanup
                .> cleanup.control  exchange=default(direct) key=cleanup.control
                .> code_owners      exchange=default(direct) key=code_owners
                .> commits          exchange=default(direct) key=commits
                .> counters-0       exchange=default(direct) key=counters-0
                .> data_export      exchange=default(direct) key=data_export
                .> default          exchange=default(direct) key=default
                .> delayed_rules    exchange=default(direct) key=delayed_rules
                .> delete_seer_grouping_records_by_hash exchange=default(direct) key=delete_seer_grouping_records_by_hash
                .> derive_code_mappings exchange=default(direct) key=derive_code_mappings
                .> digests.delivery exchange=default(direct) key=digests.delivery
                .> digests.scheduling exchange=default(direct) key=digests.scheduling
                .> dynamicsampling  exchange=default(direct) key=dynamicsampling
                .> email            exchange=default(direct) key=email
                .> email.control    exchange=default(direct) key=email.control
                .> email.inbound    exchange=default(direct) key=email.inbound
                .> events.preprocess_event exchange=default(direct) key=events.preprocess_event
                .> events.process_event exchange=default(direct) key=events.process_event
                .> events.reprocessing.preprocess_event exchange=default(direct) key=events.reprocessing.preprocess_event
                .> events.reprocessing.process_event exchange=default(direct) key=events.reprocessing.process_event
                .> events.reprocessing.symbolicate_event exchange=default(direct) key=events.reprocessing.symbolicate_event
                .> events.reprocessing.symbolicate_event_low_priority exchange=default(direct) key=events.reprocessing.symbolicate_event_low_priority
                .> events.save_event exchange=default(direct) key=events.save_event
                .> events.save_event_attachments exchange=default(direct) key=events.save_event_attachments
                .> events.save_event_highcpu exchange=default(direct) key=events.save_event_highcpu
                .> events.save_event_transaction exchange=default(direct) key=events.save_event_transaction
                .> events.symbolicate_event exchange=default(direct) key=events.symbolicate_event
                .> events.symbolicate_event_low_priority exchange=default(direct) key=events.symbolicate_event_low_priority
                .> events.symbolicate_js_event exchange=default(direct) key=events.symbolicate_js_event
                .> events.symbolicate_js_event_low_priority exchange=default(direct) key=events.symbolicate_js_event_low_priority
                .> events.symbolicate_jvm_event exchange=default(direct) key=events.symbolicate_jvm_event
                .> events.symbolicate_jvm_event_low_priority exchange=default(direct) key=events.symbolicate_jvm_event_low_priority
                .> files.copy       exchange=default(direct) key=files.copy
                .> files.delete     exchange=default(direct) key=files.delete
                .> files.delete.control exchange=default(direct) key=files.delete.control
                .> group_owners.process_commit_context exchange=default(direct) key=group_owners.process_commit_context
                .> group_owners.process_suspect_commits exchange=default(direct) key=group_owners.process_suspect_commits
                .> hybrid_cloud.control_repair exchange=default(direct) key=hybrid_cloud.control_repair
                .> incident_snapshots exchange=default(direct) key=incident_snapshots
                .> incidents        exchange=default(direct) key=incidents
                .> integrations     exchange=default(direct) key=integrations
                .> integrations.control exchange=default(direct) key=integrations.control
                .> integrations_slack_activity_notify exchange=default(direct) key=integrations_slack_activity_notify
                .> merge            exchange=default(direct) key=merge
                .> notifications    exchange=default(direct) key=notifications
                .> nudge.invite_missing_org_members exchange=default(direct) key=invite_missing_org_members
                .> on_demand_metrics exchange=default(direct) key=on_demand_metrics
                .> options          exchange=default(direct) key=options
                .> options.control  exchange=default(direct) key=options.control
                .> outbox           exchange=default(direct) key=outbox
                .> outbox.control   exchange=default(direct) key=outbox.control
                .> performance.statistical_detector exchange=default(direct) key=performance.statistical_detector
                .> post_process_errors exchange=default(direct) key=post_process_errors
                .> post_process_issue_platform exchange=default(direct) key=post_process_issue_platform
                .> post_process_transactions exchange=default(direct) key=post_process_transactions
                .> profiles.process exchange=default(direct) key=profiles.process
                .> profiling.statistical_detector exchange=default(direct) key=profiling.statistical_detector
                .> relay_config     exchange=default(direct) key=relay_config
                .> relay_config_bulk exchange=default(direct) key=relay_config_bulk
                .> releasemonitor   exchange=default(direct) key=releasemonitor
                .> relocation       exchange=default(direct) key=relocation
                .> replays.delete_replay exchange=default(direct) key=replays.delete_replay
                .> replays.ingest_replay exchange=default(direct) key=replays.ingest_replay
                .> reports.deliver  exchange=default(direct) key=reports.deliver
                .> reports.prepare  exchange=default(direct) key=reports.prepare
                .> search           exchange=default(direct) key=search
                .> sentry_metrics.indexer exchange=default(direct) key=sentry_metrics.indexer
                .> similarity.index exchange=default(direct) key=similarity.index
                .> sleep            exchange=default(direct) key=sleep
                .> stats            exchange=default(direct) key=stats
                .> subscriptions    exchange=default(direct) key=subscriptions
                .> transactions.name_clusterer exchange=default(direct) key=transactions.name_clusterer
                .> triggers-0       exchange=default(direct) key=triggers-0
                .> unmerge          exchange=default(direct) key=unmerge
                .> update           exchange=default(direct) key=update
                .> uptime           exchange=default(direct) key=uptime
                .> webhook.control  exchange=default(direct) key=webhook.control
                .> weekly_escalating_forecast exchange=default(direct) key=weekly_escalating_forecast
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_DISABLE_RATE_LIMITS' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the worker_disable_rate_limits instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_ACCEPT_CONTENT' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the accept_content instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'BROKER_TRANSPORT_OPTIONS' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the broker_transport_options instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_CREATE_MISSING_QUEUES' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_create_missing_queues instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'BROKER_URL' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the broker_url instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_DEFAULT_QUEUE' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_default_queue instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_SEND_EVENTS' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the worker_send_task_events instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_DEFAULT_EXCHANGE' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_default_exchange instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_TASK_SERIALIZER' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_serializer instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_TASK_PROTOCOL' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_protocol instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_RESULT_SERIALIZER' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_serializer instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERYBEAT_SCHEDULE' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the beat_schedule instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_TASK_RESULT_EXPIRES' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_expires instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_DEFAULT_ROUTING_KEY' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_default_routing_key instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_REDIRECT_STDOUTS' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the worker_redirect_stdouts instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_DEFAULT_EXCHANGE_TYPE' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_default_exchange_type instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERYBEAT_SCHEDULE_FILENAME' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the beat_schedule_filename instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERYD_HIJACK_ROOT_LOGGER' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the worker_hijack_root_logger instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_IGNORE_RESULT' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_ignore_result instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_QUEUES' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_queues instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_ALWAYS_EAGER' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_always_eager instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_backend instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_IMPORTS' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the imports instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/app/utils.py:203: CDeprecationWarning:
    The 'CELERY_EAGER_PROPAGATES_EXCEPTIONS' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the task_eager_propagates instead
  deprecated.warn(description=f'The {setting!r} setting',
15:01:09 [WARNING] celery.apps.worker: Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.
15:01:09 [WARNING] py.warnings: /.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(
15:02:00 [INFO] sentry.rules.delayed_processing: delayed_processing.project_id_list (project_ids='')
15:03:00 [INFO] sentry.rules.delayed_processing: delayed_processing.project_id_list (project_ids='')
worker: Warm shutdown (MainProcess)
15:03:36 [INFO] sentry.bgtasks: bgtask.stop (task_name='sentry.bgtasks.clean_dsymcache:clean_dsymcache')
15:03:36 [INFO] sentry.bgtasks: bgtask.stop (task_name='sentry.bgtasks.clean_releasefilecache:clean_releasefilecache')

[hanife-Orthopy]

After increasing the count of worker replicas to 2 and, if still necessary, increasing the CPU and memory limits, the sentry-worker started functioning properly.

[patsevanton]

@theechofive have you checked if sentry is installed without agrocd?