d6c72d19e6
Updates to Airflow 1.10.1; See (1), (2) for some notes Related, and additionally: configures Airflow to restore logging of workflow steps to a console/sdtout logger, supporting the desired ability to attach logging and monitoring to standard container mechanisms. This does not change the behavior of also logging to the airflow-arranged log files for steps and DAG runs. A side effect of updating to 1.10.1 includes a major decrease in resource usage by the Airflow scheudler process (reducing from ~ 1 core fully consumed to less than 5% of a core consumed YMMV, but significant) Additional adjustment downward of resources allocated, threads produced, and frequency of polling leads to an overall significant reduction in resource usage. Airship note: Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and dag_run information, updating from 1.10.0 - 1.10.1 in place is possible if airflow-worker pods are allowed to continue to run. (1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md (2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1 Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04
0 lines
Python
0 lines
Python