Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dag blocked when winter time change happen with specific cron #35558

Closed
2 tasks done
V0lantis opened this issue Nov 9, 2023 · 2 comments
Closed
2 tasks done

Dag blocked when winter time change happen with specific cron #35558

V0lantis opened this issue Nov 9, 2023 · 2 comments
Labels
area:core area:Scheduler including HA (high availability) scheduler kind:bug This is a clearly a bug

Comments

@V0lantis
Copy link
Contributor

V0lantis commented Nov 9, 2023

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

After the time shift which happen in Europe during the last 29th morning of October, we noticed that some of our dag we specific cron timetable were not scheduling anymore.

The dag definition is as follow:

with DAG(
    dag_id="dag_id",
    schedule="0 7-21 * * *",
    max_active_runs=1,
) as dag:

with the following start_date

start_date = timezone.parse("2022-04-26", "Europe/Paris")

What you think should happen instead

The scheduler should have been able to continue scheduling new dag run normally.

How to reproduce

I created the following tests, to be able to dig into the issue. I will try to implement a better one soon

from __future__ import annotations

from pendulum import DateTime
from pendulum.tz.timezone import UTC

from airflow.timetables._cron import CronMixin

class TestCronMixin:
    def test_get_next_true(self):
        cron_mixin = CronMixin("0 7-21 * * *", timezone="Europe/Paris")
        print("\n")
        for day, hour in [(28, 20), (28, 21), (28, 22), (28, 23), (29, 0), (29, 1), (29, 2), (29, 3), (29, 4), (29, 5), (29, 6), (29, 7)]:
            date = DateTime(year=2023, month=10, day=day, hour=hour, minute=0, second=0, tzinfo=UTC)
            align_prev_date = cron_mixin._align_to_prev(date)
            print(f"the {day} at {hour}: pre_date={align_prev_date}")

Operating System

NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" SUPPORT_END="2025-06-30"

Versions of Apache Airflow Providers

$ pip freeze | grep providers
apache-airflow-providers-amazon==8.3.1
apache-airflow-providers-celery==3.2.1
apache-airflow-providers-cncf-kubernetes==7.4.2
apache-airflow-providers-common-sql==1.6.0
apache-airflow-providers-datadog==3.3.1
apache-airflow-providers-docker==3.7.1
apache-airflow-providers-ftp==3.4.2
apache-airflow-providers-github==2.3.1
apache-airflow-providers-google==10.3.0
apache-airflow-providers-hashicorp==3.5.0
apache-airflow-providers-http==4.4.2
apache-airflow-providers-imap==3.2.2
apache-airflow-providers-mysql==5.1.1
apache-airflow-providers-postgres==5.5.2
apache-airflow-providers-redis==3.4.0
apache-airflow-providers-salesforce==5.4.1
apache-airflow-providers-sftp==4.3.1
apache-airflow-providers-slack==7.3.1
apache-airflow-providers-sqlite==3.5.0
apache-airflow-providers-ssh==3.7.1
apache-airflow-providers-tableau==4.2.1
apache-airflow-providers-zendesk==4.3.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

Kubernetes deployment

Anything else

This problem will probably occurs every time we will experience the winter time shift.

Luckily, we mitigate the issue by putting catchup=False to those dags which didn't needed to catchup (obviously), but if we did need to have a proper history for every hour, I don't know how we could have deal with this.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@V0lantis V0lantis added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Nov 9, 2023
@V0lantis
Copy link
Contributor Author

V0lantis commented Nov 9, 2023

After a lot of investigation, I am pretty sure the problem comes from the following snippet:

    def _align_to_prev(self, current: DateTime) -> DateTime:
        """Get the prev scheduled time.

        This is ``current - interval``, unless ``current`` falls right on the
        interval boundary, when ``current`` is returned.
        """
        prev_time = self._get_prev(current)
        if self._get_next(prev_time) != current:
            return prev_time
        return current

Indeed, the test that I wrote above give us the following result:

the 28 at 20 UTC: naive=2023-10-28 22:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T19:00:00+00:00
the 28 at 21 UTC: naive=2023-10-28 23:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T19:00:00+00:00
the 28 at 22 UTC: naive=2023-10-29 00:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T19:00:00+00:00
the 28 at 23 UTC: naive=2023-10-29 01:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T19:00:00+00:00
the 29 at 0 UTC: naive=2023-10-29 02:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T19:00:00+00:00
the 29 at 1 UTC: naive=2023-10-29 02:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T20:00:00+00:00 # Same from midnight
the 29 at 2 UTC: naive=2023-10-29 03:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T20:00:00+00:00
the 29 at 3 UTC: naive=2023-10-29 04:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T20:00:00+00:00
the 29 at 4 UTC: naive=2023-10-29 05:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T20:00:00+00:00
the 29 at 5 UTC: naive=2023-10-29 06:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-29T05:00:00+00:00
the 29 at 6 UTC: naive=2023-10-29 07:00:00 | scheduled=2023-10-28 21:00:00 | pre_date=2023-10-28T20:00:00+00:00
the 29 at 7 UTC: naive=2023-10-29 08:00:00 | scheduled=2023-10-29 07:00:00 | pre_date=2023-10-29T07:00:00+00:00

@V0lantis V0lantis changed the title Dag blocked when winter time change happen Dag blocked when winter time change happen with specific cron Nov 9, 2023
@hussein-awala hussein-awala added area:Scheduler including HA (high availability) scheduler and removed needs-triage label for new issues that we didn't triage yet labels Nov 9, 2023
@V0lantis
Copy link
Contributor Author

V0lantis commented Jan 9, 2024

I think this is fixed by #35887

@V0lantis V0lantis closed this as completed Jan 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core area:Scheduler including HA (high availability) scheduler kind:bug This is a clearly a bug
Projects
None yet
Development

No branches or pull requests

2 participants