Airflow on CF: CeleryExecutor/redis communication issues

[robert-bryson]

Airflow can use a variety of executors. Should we want to use the deploy scenario of Airflow on Cloud Foundry, we would need to fix the CeleryExecutor. In the current deployment (https://test-airflow-webserver.app.cloud.gov/), there is an issue somewhere around the communication of the application components, the celery workers, and redis.

See additional context #4434 and GSA/datagov-harvester#1.

How to reproduce

1. Inspect the test-airflow-redis service and various components of the current test deployment:
   
2. See that they are not communicating correctly in flower: https://test-airflow-flower.app.cloud.gov/

Expected behavior

Something akin to this:

Actual behavior

No workers :/

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

• Not sure what would be useful here, I have been experimenting with using the docker-compose.yml version of the deploy locally and contrasting that with the config/env on the cf deploy

[hkdctol]

Icebox for now, because currently not planning on using Celery Executor

[btylerburton]

Rudimentary configuration of CeleryExecutor with Webserver, Scheduler and Worker is complete.

We will add additional components (Flower, Triggerer, DagProcessor) at a later date, but communication issues have been solved.