Error in web service after chart version 25.20.0 (sentry 24.8.0) #1628

piradata · 2024-12-10T00:00:16Z

Issue submitter TODO list

I've searched for an already existing issues here

Describe the bug (actual behavior)

we tried to upgrade from 25.20.0 (sentry 24.8.0) to chart 26.7.0 (sentry 24.9.0) and the web server keeps restarting with the following error while all the others services seem to be ok.

Tried with others versions on 26.x.x and got the same error (except version 26.8.0 that has another error about sidecar, issue #1624).

Expected behavior

Update should work

values.yaml

# Command to run:
# helm -n sentry upgrade --force sentry sentry/sentry --wait --timeout=1000s --version 26.8.0 -f sentry/sentry.yaml

# Set this to true to support IPV6 networks
# ipv6: false

user:
  create: false

# this is required on the first installation, as sentry has to be initialized first
# recommended to set false for updating the helm chart afterwards,
# as you will have some downtime on each update if it's a hook
# deploys relay & snuba consumers as post hooks
asHook: false

hooks:
  dbInit:
    enabled: true
  snubaInit:
    enabled: true
  snubaMigrate:
    enabled: true

serviceAccount:
  # serviceAccount.annotations -- Additional Service Account annotations.
  annotations: {}
  # serviceAccount.enabled -- If `true`, a custom Service Account will be used.
  enabled: false
  # serviceAccount.name -- The base name of the ServiceAccount to use. Will be appended with e.g. `snuba-api` or `web` for the pods accordingly.
  name: "sentry"
  # serviceAccount.automountServiceAccountToken -- Automount API credentials for a Service Account.
  automountServiceAccountToken: true

sentry:
  # to not generate a sentry-secret, use these 2 values to reference an existing secret
  # existingSecret: "my-secret"
  # existingSecretKey: "my-secret-key"
  singleOrganization: true
  web:
    enabled: true
    # if using filestore backend filesystem with RWO access, set strategyType to Recreate
    strategyType: RollingUpdate
    replicas: 1
    env: []
    probeFailureThreshold: 5
    probeInitialDelaySeconds: 10
    probePeriodSeconds: 10
    probeSuccessThreshold: 1
    probeTimeoutSeconds: 2
    resources: {}
    affinity: {}
    nodeSelector: {}
    securityContext: {}
    # if you are using GKE Ingress controller use 'securityPolicy' to add Google Cloud Armor Ingress policy
    securityPolicy: ""
    # if you are using GKE Ingress controller use 'customResponseHeaders' to add custom response header
    customResponseHeaders: []
    containerSecurityContext: {}
    service:
      annotations: {}
    # tolerations: []
    # podLabels: []

    autoscaling:
      enabled: false
      minReplicas: 2
      maxReplicas: 5
      targetCPUUtilizationPercentage: 50
    sidecars: []
    topologySpreadConstraints: []
    volumes: []
    volumeMounts: []

  features:
    orgSubdomains: false
    vstsLimitedScopes: true
    enableProfiling: true
    enableSessionReplay: true
    enableFeedback: false
    enableSpan: false

  # example customFeature to enable Metrics(beta) https://docs.sentry.io/product/metrics/
  # customFeatures:
  # - organizations:custom-metric
  # - organizations:custom-metrics-experimental
  # - organizations:derive-code-mappings

  cleanup:
    successfulJobsHistoryLimit: 5
    failedJobsHistoryLimit: 5
    activeDeadlineSeconds: 100
    concurrencyPolicy: Allow
    concurrency: 1
    enabled: true
    schedule: "0 0 * * *"
    days: 60
    # logLevel: INFO
    logLevel: ''
    # securityContext: {}
    # containerSecurityContext: {}
    sidecars: []
    volumes: []
    # volumeMounts: []
    serviceAccount: {}

mail:
  backend: smtp
  useTls: true
  useSsl: false
  username: "XXXXXXXXXXXXXXXXXXXXXX"
  existingSecret: sentry-smtp-secrets
  existingSecretKey: SMTP_PASSWORD
  port: 25
  host: "email-smtp-enterprize.us-east-1.amazonaws.com"
  from: "[email protected]"

auth:
  register: false

service:
  name: sentry
  type: ClusterIP
  externalPort: 9000
  annotations: {}

nginx:
  enabled: false

ingress:
  enabled: true
  regexPathStyle: nginx
  ingressClassName: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/proxy-body-size: 10m
    nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    nginx.ingress.kubernetes.io/session-cookie-name: Keycloak-Node-Id
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hostname: sentry.enterprize.com
  tls:
  - secretName: sentry-enterprize-tls-cert
    hosts: 
    - sentry.enterprize.com

symbolicator:
  enabled: true

sourcemaps:
  enabled: true

filestore:
  backend: s3
  s3:
   existingSecret: sentry-s3-secrets
   accessKeyIdRef: ACCESS_KEY
   secretAccessKeyRef: SECRET_KEY
   bucketName: enterprize-sentry-files
   region_name: us-east-1
   default_acl: private

# Reference -> https://docs.sentry.io/product/integrations/source-code-mgmt/github/
github:
  appId: "986340"
  appName: Sentry Enterprize
  existingSecret: "sentry-github-app-secrets"
  existingSecretPrivateKeyKey: "private-key"
  existingSecretWebhookSecretKey: "webhook-secret"
  existingSecretClientIdKey: "client-id"
  existingSecretClientSecretKey: "client-secret"

kafka:
  enabled: true
  zookeeper:
    enabled: false
  kraft:
    enabled: true
  controller:
    replicaCount: 3
    ## if the load on the kafka controller increases, resourcesPreset must be increased
    resourcesPreset: medium # small, medium, large, xlarge, 2xlarge
    controllerOnly: false
    persistence:
      size: 60Gi
  extraConfigYaml:
    log.retention.hours: 24
    log.retention.bytes: 53687091200
    log.segment.bytes: 1073741824
    log.retention.check.interval.ms: 300000
    log.segment.delete.delay.ms: 60000
  provisioning:
    enabled: true
    topics:
      - name: events
        config:
          "message.timestamp.type": LogAppendTime
      - name: event-replacements
      - name: snuba-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: cdc
      - name: transactions
        config:
          "message.timestamp.type": LogAppendTime
      - name: snuba-transactions-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: snuba-metrics
        config:
          "message.timestamp.type": LogAppendTime
      - name: outcomes
      - name: outcomes-billing
      - name: ingest-sessions
      - name: snuba-sessions-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: snuba-metrics-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: scheduled-subscriptions-events
      - name: scheduled-subscriptions-transactions
      - name: scheduled-subscriptions-sessions
      - name: scheduled-subscriptions-metrics
      - name: scheduled-subscriptions-generic-metrics-sets
      - name: scheduled-subscriptions-generic-metrics-distributions
      - name: scheduled-subscriptions-generic-metrics-counters
      - name: events-subscription-results
      - name: transactions-subscription-results
      - name: sessions-subscription-results
      - name: metrics-subscription-results
      - name: generic-metrics-subscription-results
      - name: snuba-queries
        config:
          "message.timestamp.type": LogAppendTime
      - name: processed-profiles
        config:
          "message.timestamp.type": LogAppendTime
      - name: profiles-call-tree
      - name: ingest-replay-events
        config:
          "message.timestamp.type": LogAppendTime
          "max.message.bytes": "15000000"
      - name: snuba-generic-metrics
        config:
          "message.timestamp.type": LogAppendTime
      - name: snuba-generic-metrics-sets-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: snuba-generic-metrics-distributions-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: snuba-generic-metrics-counters-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: generic-events
        config:
          "message.timestamp.type": LogAppendTime
      - name: snuba-generic-events-commit-log
        config:
          "cleanup.policy": "compact,delete"
          "min.compaction.lag.ms": "3600000"
      - name: group-attributes
        config:
          "message.timestamp.type": LogAppendTime
      - name: snuba-attribution
      - name: snuba-dead-letter-metrics
      - name: snuba-dead-letter-sessions
      - name: snuba-dead-letter-generic-metrics
      - name: snuba-dead-letter-replays
      - name: snuba-dead-letter-generic-events
      - name: snuba-dead-letter-querylog
      - name: snuba-dead-letter-group-attributes
      - name: ingest-attachments
      - name: ingest-transactions
      - name: ingest-events
      - name: ingest-replay-recordings
      - name: ingest-metrics
      - name: ingest-performance-metrics
      - name: ingest-monitors
      - name: profiles
      - name: ingest-occurrences
      - name: snuba-spans
      - name: shared-resources-usage
      - name: snuba-metrics-summaries
  listeners:
    client:
      protocol: "PLAINTEXT"
    controller:
      protocol: "PLAINTEXT"
    interbroker:
      protocol: "PLAINTEXT"
    external:
      protocol: "PLAINTEXT"

zookeeper:
  enabled: true
  replicaCount: 1
  persistence:
    size: 30Gi
  autopurge:
    purgeInterval: 6
    snapRetainCount: 3

postgresql:
  enabled: false

externalPostgresql:
  host: keycloak-qa-db.clxbcprct36q.us-east-1.rds.amazonaws.com
  port: 5432
  database: sentry
  existingSecret: sentry-db-secrets
  existingSecretKeys:
    password: DB_PASSWORD
    username: DB_USER
  ## Default connection max age is 0 (unlimited connections)
  ## Set to a higher number to close connections after a period of time in seconds
  connMaxAge: 0

## Prometheus Exporter / Metrics
##
metrics:
  enabled: false

redis:
  enabled: true
  replica:
    replicaCount: 1
    resources:
      requests:
        memory: 6Gi
        cpu: 100m
      limits:
        memory: 10Gi
        cpu: 500m
  master:
    persistence:
      enabled: true
    resources:
      requests:
        memory: 6Gi
        cpu: 100m
      limits:
        memory: 10Gi
        cpu: 500m

Helm chart version

25.20.0 --> 26.8.0

Steps to reproduce

we tried to upgrade from 25.20.0 (sentry 24.8.0) to chart 26.7.0 (sentry 24.9.0) and the web server keeps restarting with the following error while all the others services seem to be ok.

Screenshots

Logs

File "/usr/src/sentry/src/sentry/runner/init .py", line 33, in configure
from sentry.utils.sdk import configure_sdk
from django.core.handlers.asgi import ASGIRequest
File "/usr/src/sentry/src/sentry/utils/sdk.py", line 26, in <module>
File "/.venv/lib/python3.12/site-packages/django/core/handlers/asgi.py", line 13, in <module>
_configure(ctx, py, yaml, skip_service_validation)
File "/usr/src/sentry/src/sentry/runner/settings.py", line 121, in configure
from django.core.handlers import base
from sentry.features.rollout import in_random_rollout
File "/.venv/lib/python3.12/site-packages/django/core/handlers/base.py", line 15, in <module>
File "/usr/src/sentry/src/sentry/features/init .py", line 1, in <module>
initialize_app(
File "/usr/src/sentry/src/sentry/runner/initializer.py", line 366, in initialize_app
from sentry.features.permanent import register_permanent_features
from .exception import convert_exception_to_response
File "/.venv/lib/python3.12/site-packages/django/core/handlers/exception.py", line 21, in <module>
File "/usr/src/sentry/src/sentry/features/permanent.py", line 7, in <module>
from sentry.features.manager import FeatureManager
from django.views import debug
File "/.venv/lib/python3.12/site-packages/django/views/debug.py", line 21, in <module>
File "/usr/src/sentry/src/sentry/features/manager.py", line 16, in <module>
django.setup()
File "/.venv/lib/python3.12/site-packages/django/init .py", line 24, in setup
from django.views.decorators.debug import coroutine_functions_to_sensitive_variables
from sentry.users.services.user.model import RpcUser
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "/usr/src/sentry/src/sentry/users/services/user/init .py", line 1, in <module>
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 921, in _load_unlocked
File "<frozen importlib._bootstrap>", line 819, in module_from_spec
apps.populate(settings.INSTALLED_APPS)
File "<frozen importlib._bootstrap>", line 798, in _init_module_attrs
File "<frozen importlib._bootstrap>", line 638, in cached
File "<frozen importlib._bootstrap_external>", line 614, in _get_cached
File "<frozen importlib._bootstrap_external>", line 513, in cache_from_source
KeyboardInterrupt
File "/.venv/lib/python3.12/site-packages/django/apps/registry.py", line 91, in populate
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. GAME OVER ***
from .model import * # noqa
^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/users/services/user/model.py", line 10, in <module>
app_config = AppConfig.create(entry)
^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.12/site-packages/django/apps/config.py", line 111, in create
from pydantic.fields import Field
File "pydantic/init .py", line 2, in init pydantic.init
app_module = import_module(entry)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/init .py", line 90, in import_module
gateway "uWSGI http 1" has been buried (pid: 11)
return _bootstrap._gcd_import(name[level:], package, level)
File "pydantic/dataclasses.py", line 52, in init pydantic.dataclasses
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/sentry/src/sentry/analytics/events/init .py", line 46, in <module>
from .issue_escalating import * # noqa: F401,F403
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1357, in _find_and_load
File "<frozen importlib._bootstrap>", line 418, in enter
File "<frozen importlib._bootstrap>", line 311, in acquire
File "<frozen importlib._bootstrap>", line 170, in enter
File "<frozen importlib._bootstrap>", line 132, in setdefault
File "<frozen importlib._bootstrap>", line 74, in new
KeyboardInterrupt
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. GAME OVER ***
# init
File "pydantic/error_wrappers.py", line 4, in init pydantic.error_wrappers
File "pydantic/json.py", line 13, in init pydantic.json
File "pydantic/networks.py", line 32, in init pydantic.networks
File "pydantic/validators.py", line 549, in init pydantic.validators
File "/usr/local/lib/python3.12/typing.py", line 1531, in getitem
return self.getitem_inner (params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/typing.py", line 395, in inner
return _caches[func](*args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/typing.py", line 1544, in getitem_inner
return self.copy_with(params)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/typing.py", line 1516, in copy_with
return _CallableGenericAlias(self.origin , params,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/typing.py", line 1248, in init
self.parameters = _collect_parameters(args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.12/site-packages/typing_extensions.py", line 3045, in _collect_parameters
elif hasattr(t, 'typing_subst '):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. GAME OVER ***
OOPS ! failed loading app in worker 1 (pid 8)
Mon Dec 9 23:24:26 2024 - need-app requested, destroying the instance...
OOPS ! failed loading app in worker 2 (pid 9)
Mon Dec 9 23:24:26 2024 - need-app requested, destroying the instance...
OOPS ! failed loading app in worker 3 (pid 10)
Mon Dec 9 23:24:26 2024 - need-app requested, destroying the instance...
goodbye to uWSGI.
Stream closed EOF for sentry/sentry-web-6dcd8cdf6-8rmpn (sentry-web)

Additional context

No response

The text was updated successfully, but these errors were encountered:

piradata · 2024-12-10T00:00:50Z

@Markgabe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error in web service after chart version 25.20.0 (sentry 24.8.0) #1628

Error in web service after chart version 25.20.0 (sentry 24.8.0) #1628

piradata commented Dec 10, 2024

piradata commented Dec 10, 2024

Error in web service after chart version 25.20.0 (sentry 24.8.0) #1628

Error in web service after chart version 25.20.0 (sentry 24.8.0) #1628

Comments

piradata commented Dec 10, 2024

Issue submitter TODO list

Describe the bug (actual behavior)

Expected behavior

values.yaml

Helm chart version

Steps to reproduce

Screenshots

Logs

Additional context

piradata commented Dec 10, 2024