Django celery results in combination with dango tenants intermittent error of table not found

[saichander17]

Hi,
We are utilizing Django Celery Results in conjunction with Django Tenants in our project. However, our Celery worker intermittently encounters an error stating that the "django_celery_results" relation cannot be found in the database. Strangely, this issue does not occur consistently, and we have verified that the table indeed exists. It's worth noting that we currently have only one tenant.

We are seeking assistance in understanding and resolving this issue. If anyone has encountered this problem before, we would greatly appreciate your insights.

Below is the error trace for reference:

[2023-07-28 07:26:50,019: ERROR/MainProcess] Process 'ForkPoolWorker-291' pid:8099 exited with 'signal 9 (SIGKILL)'
[2023-07-28 07:26:50,060: ERROR/MainProcess] Pool callback raised exception: ProgrammingError('relation "django_celery_results_taskresult" does not exist\nLINE 1: ...", "django_celery_results_taskresult"."meta" FROM "django_ce...\n ^\n')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) Job: 308397.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "django_celery_results_taskresult" does not exist
LINE 1: ...", "django_celery_results_taskresult"."meta" FROM "django_ce...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/billiard/pool.py", line 1796, in safe_apply_callback
fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/worker/request.py", line 577, in on_failure
self.task.backend.mark_as_failure(
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 172, in mark_as_failure
self.store_result(task_id, exc, state,
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 528, in store_result
self._store_result(task_id, result, state, traceback,
File "/usr/local/lib/python3.8/site-packages/django_celery_results/backends/database.py", line 66, in _store_result
self.TaskModel._default_manager.store_result(
File "/usr/local/lib/python3.8/site-packages/django_celery_results/managers.py", line 46, in _inner
return fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django_celery_results/managers.py", line 168, in store_result
obj, created = self.using(using).get_or_create(task_id=task_id,
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 581, in get_or_create
return self.get(**kwargs), False
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
num = len(clone)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
self._fetch_all()
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1169, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
return super().execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "django_celery_results_taskresult" does not exist
LINE 1: ...", "django_celery_results_taskresult"."meta" FROM "django_ce...
^

[auvipy]

this could be a schema migrations issue in you project and DB

[saichander17]

@auvipy , Thanks for your response
If it's related to table not being present really, it should fail everytime right?
After spending sometime on the trace, I think but not sure that it is workerlost error is coming up and whenever task fails, it's not able to set the correct schema which is resulting in table not found error. I'm not sure about why the worker lost error is coming though!