Dag blocked when winter time change happen with specific cron #35558
Labels
area:core
area:Scheduler
including HA (high availability) scheduler
kind:bug
This is a clearly a bug
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
After the time shift which happen in Europe during the last 29th morning of October, we noticed that some of our dag we specific cron timetable were not scheduling anymore.
The dag definition is as follow:
with the following
start_date
What you think should happen instead
The scheduler should have been able to continue scheduling new dag run normally.
How to reproduce
I created the following tests, to be able to dig into the issue. I will try to implement a better one soon
Operating System
NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" SUPPORT_END="2025-06-30"
Versions of Apache Airflow Providers
$ pip freeze | grep providers apache-airflow-providers-amazon==8.3.1 apache-airflow-providers-celery==3.2.1 apache-airflow-providers-cncf-kubernetes==7.4.2 apache-airflow-providers-common-sql==1.6.0 apache-airflow-providers-datadog==3.3.1 apache-airflow-providers-docker==3.7.1 apache-airflow-providers-ftp==3.4.2 apache-airflow-providers-github==2.3.1 apache-airflow-providers-google==10.3.0 apache-airflow-providers-hashicorp==3.5.0 apache-airflow-providers-http==4.4.2 apache-airflow-providers-imap==3.2.2 apache-airflow-providers-mysql==5.1.1 apache-airflow-providers-postgres==5.5.2 apache-airflow-providers-redis==3.4.0 apache-airflow-providers-salesforce==5.4.1 apache-airflow-providers-sftp==4.3.1 apache-airflow-providers-slack==7.3.1 apache-airflow-providers-sqlite==3.5.0 apache-airflow-providers-ssh==3.7.1 apache-airflow-providers-tableau==4.2.1 apache-airflow-providers-zendesk==4.3.1
Deployment
Official Apache Airflow Helm Chart
Deployment details
Kubernetes deployment
Anything else
This problem will probably occurs every time we will experience the winter time shift.
Luckily, we mitigate the issue by putting
catchup=False
to those dags which didn't needed to catchup (obviously), but if we did need to have a proper history for every hour, I don't know how we could have deal with this.Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: